nL-E  COPY 


AFHRL-TR-90-2 


AIR  FORCE  B 


SUPERKEATS:  A  SYSTEM  TO  SUPPORT  TRAINING 
FOR  DECISION-MAKING  SKILLS 


Fritz  H.  Brecke 
Patrick  J.  Hays 
Donald  L.  Johnston 
Jane  M.  McGarvey 
Susan  M.  Peters 
Gail  K.  Siemon 


DT1C 

ft  ELECTE^ 
V  MAY  3 11990 1  1 


Log  icon,  Incorporated 
Tactical  and  Training  Systems  Division 
4010  Sorrento  Valley  Boulevard 
San  Diego,  California  92138-5158 

LOGISTICS  AND  HUMAN  FACTORS  DIVISION 
Wright-Patterson  Air  Force  Base,  Ohio  45433-6503 


May  1990 

Final  Technical  Report  for  Period  November  1986  -  January  1990 


Approved  for  public  release;  distribution  is  unlimited. 


LABORATORY 


AIR  FORCE  SYSTEMS  COMMAND 
BROOKS  AIR  FORCE  BASE,  TEXAS  7S235-5601 


ft  V 


NOTICE 


When  Government  drawings,  specifications,  or  other  data  are  used  for  any  purpose 
other  than  in  connection  with  a  definitely  Government-related  procurement,  the 
United  States  Government  incurs  no  responsibility  or  any  obligation  whatsoever. 
The  fact  that  the  Government  may  have  formulated  or  in  any  way  supplied  the  said 
drawings,  specifications,  or  other  data,  is  not  to  be  regarded  by  implication,  or 
otherwise  in  any  manner  construed,  as  licensing  the  holder,  or  any  other  person  or 
corporation;  or  as  conveying  any  rights  or  permission  to  manufacture,  use,  or  sell 
any  patented  invention  that  may  in  any  way  be  related  thereto. 

The  Public  Affairs  Office  has  reviewed  this  report,  and  it  is  releasable  to  the  National 
Technical  Information  Service,  where  it  will  be  available  to  the  general  public, 
including  foreign  nationals. 

This  report  has  been  reviewed  and  is  approved  for  publication. 


BERTRAM  W.  CREAM,  Technical  Director 
Logistics  and  Human  Factors  Division 


HAROLD  G.  JENSEN,  Colonel,  USAF 
Commander 


REPORT  DOCUMENTATION  PAGE 

Form  approved 

OMS  NO.  0704-019$ 

Put**  reporting  burboo  for  thn  cottecbon  of  information  *»  estimated  to  mf«9»  (  Hour  otf  rnpom*.  mdudfrig  th«  tim«  for  r*v«wmg  tmtrucuom.  laorthfng  erttonq  data  lourcaa, 
oatfwrmq  and  mamtatmog  tha  data  madid,  and  comptamg  and  r*v*awing  tha  <o1t*ci>oo  of  information  Sand  commants  rvaardtog  thn  bordan  asomaca  or  any  Othar  npaa  of  «M» 
coUactwn  of  information,  including  wggautons  for  raduemg  thtt  burdan.  to  Washington  Haadouartars  SarvKas.  Oiractorata  For  information  Oparabona  and  Mpom.  1211  JaffaHoa 
Oam  Highway.  Suita  1204.  Arlington.  VA  22202 -4 M2,  and  to  tha  Offlca  of  Managamant  and  Budgat.  Paparwork  Reduction  Profact  (0704-0 IBB).  Washington.  OC  20901. 

T  AGENCY  USE  ONLY  (Leave  blink)  2.  REPORT  DATE  1.  REPORT  TYPE  AND  OATES  COVERED 

May  1990  Final  -  November  1986  -  January  1990 

4.  TITLE  AND  SUBTITLE 

SUPERKEATS:  A  System  to  Support  Training 
for  Decision-Making  Skills 

5.  FUNDING  NUMBERS 

C  -  F3361 5-86-C-0020 

PE  -  62205F 

PR  -  3017 

TA  -  08 

WU  -  15 

6.  AUTHORS) 

Fritz  H.  Brecke  Jane  M.  McGarvey 

Patrick  J.  Hays  Susan  M.  Peters 

Donald  L.  Johnston  Gail  K.  Slemon 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADORE  SS(ES) 

Eogicon,  Incorporated 

Tactical  and  Training  Systems  Division 

4010  Sorrento  Valley  Boulevard 

San  Diego,  California  92138-5158 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING /MONITORING  AGENCY  NAME(S)  AND  AOORESS(ES) 

Logistics  and  Human  Factors  Division 

Air  Force  Human  Resources  Laboratory 

Wright-Patterson  Air  Force  Base,  Ohio  45433-6503 

10.  SPONSORING /MONITORING 

AGENCY  REPORT  NUMBER 

AFHRL-TR-90-2 

11.  SUPPLEMENTARY  NOTES 

12a.  DISTRIBUTION /AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  is  unlimited. 

12b.  DISTRIBUTION  CODE 

13.  ABSTRACT  (Maximum  200  words) 

‘  i -Training  in  decision-making  skills  contributes  directly  to  the  combat  readiness  of  battle 
managers.  More  affordable,  accessible,  and  effective  technologies  are  required  to  supplement  the 

Command  Post  Exercise  technologies  currently  In  use.  This  report  covers  the  third  phase  of  a  project 
aimed  toward  the  development  of  such  new  technologies.  During  the  preceding  phases,  a  Knowledge 

Engineering  and  Training  System  (KEATS)  was  developed  and  used  to  acquire  task  knowledge  employed  by 
experts  during  decision-making  In  a  specific  tactical  domain.  In  the  third  and  final  phase  of  the 
project,  the  acquired  knowledge  was  used  to  develop  a  practice  environment  for  decision-making.  The 
practice  environment,  called  SuperKEATS,  Is  an  object-oriented,  simulated  practice  module  written  in 

KEE  which  runs  on  workstations.  SuperKEATS  generates  a  functionally  realistic,  reactive  exercise 
scenario  where  decisions  produce  plausible,  natural  feedback  as  well  as  explanatory  artificial 
feedback.  Optional  prompts  provide  general  suggestions  and/or  specific  options.  SuperKEATS  was 
subjected  to  a  2-day  evaluation  trial  with  one  expert  and  one  novice.  The  trial  provided  a  successful 
demonstration  of  the  concepts  of  a  responsive  system  for  decision  training.  The  results  are 
described,  and  conclusions  and  recommendations  are  offered.  \sr,  ,  / 

i  d  ^ 

14.  SUBJECT  TERMS 

; »  computer-based  training ^  knowledge  acquisition/  tactical  command^ 

decision-making  training »  knowledge  engineering,  and  control*  f'S(\,\\ 

expert  knowledge  micro-computer/  V 

IS.  NUMBEB  OP  PAGES 

92 

IS.  PRICE  COOf 

17.  SECURITY  CLASSIFICATION 
Of  REPORT 

Unclassified 

If.  SECURITY  CLASSIFICATION 
Of  THIS  PAGE 

Unclassified 

1*.  SECURITY  CLASSHCATKJN 
Of  ABSTRACT 

Unclassified 

20.  LIMITATION  OF  ABSTRACT 

UL 

NSN  7540*01-280-5500  Standard  Form  298  <R#v  2-89) 


ar«%cr«Cd  by  ANSI  tM  2J9-’B 


SUMMARY 


Battle  managers  assigned  to  Air  Force  Tactical  Air  Control  Systems  must  be  competent 
decision-makers.  Improved  technologies  are  required  to  develop  and  deliver  training  in  decision-making 
skills  at  an  affordable  cost.  Training  should  be  available  at  peacetime  duty  stations,  run  on 
microcomputers  available  at  the  squadron  level,  and  not  require  instructors. 

A  preceding  report  described  a  technology  for  acquiring  the  instructional  content  for  a  target  decision 
training  task.  A  computer-based  (Smalltalk  and  PC/AT)  Knowledge  Engineering  and  Training  System 
(KEATS)  (Brecke  et  al.,  1989)  was  developed  and  applied  to  a  specific  tactical  decision  domain.  KEATS 
helped  identify  schemas  for  situation  assessment  and  plans,  and  decision  rules  for  tasking  of  air  requests. 

The  present  report  focuses  on  the  development  of  a  practice  environment  for  decision-making.  This 
environment,  called  SuperKEATS,  was  implemented  as  a  stand-alone  prototype  with  content  previously 
acquired  through  the  use  of  KEATS.  The  report  describes  a  2-day,  two-subject  preliminary  evaluation  of 
SuperKEATS  and  ends  with  a  discussion  and  conclusions  regarding  the  viability  of  the  training 
methodology  represented  by  SuperKEATS. 
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PREFACE 


With  its  establishment  in  1980,  the  Logistics  and  Human  Factors  Division’s  Ground  Operations 
Branch,  a  part  of  the  Air  Force  Human  Resources  Laboratory  (AFHRL/LRG)  at  Wright-Patterson  AFB, 
began  a  comprehensive  program  to  develop  advanced  technology  to  improve  training  for  USAF  Tactical 
Command  and  Control  (C2)  Battle  Staff  personnel. 

This  report  covers  the  final  phase  of  a  research  project  which  developed  methods  and  principles  for 
cost-effective  tactical  C 2  decision-making  training.  A  previous  phase  centered  around  the  use  of 
knowledge  engineering  techniques  to  acquire  instructional  content  by  means  of  a  Knowledge  Engineering 
and  Training  System  (KEATS).  The  system  was  effective  in  generating  the  decision  task  knowledge 
needed  for  the  final  phase  of  the  project.  During  the  final  phase,  a  practice  environment  for  tactical 
decision-making,  called  SuperKEATS,  was  implemented  as  a  stand-alone  prototype.  The  system 
incorporated  the  decision  task  knowledge  previously  acquired  through  the  use  of  KEATS. 

The  project  was  guided  by  Mr.  Michael  J.  Young  of  AFHRL/LRG  and  executed  by  Logicon’s  Tactical 
and  Training  Systems  Division  in  San  Diego,  California,  under  the  direction  of  Fritz  H.  Brecke.  Logicon 
team  members  included:  Patrick  Hays,  Donald  Johnston,  Gail  Slemon,  Jane  McGarvey,  and  Susan 
Peters.  Particularly  heartfelt  thanks  go  to  the  Air  Force  subject-matter  experts:  Lt  Col  Fred  Wilson,  Lt 
Col  Lynn  Weber,  Lt  Col  Matt  Szczepanek,  and  Maj  Ken  Dekay. 
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I.  PROBLEM 


Project  Background  and  Objectives 

Rapid  and  accurate  decision-making  under  time  constraints,  combat  stress  and  the  uncertainty  of  the 
“fog  of  war”  is  a  critical  combat  skill.  High  levels  of  proficiency  in  decision-making  are  particularly 
important  for  tactical  battle  staff  officers  assigned  to  Tactical  Command  and  Control  (TC2)  systems  in  the 
military  services.  These  officers  manage  high-value  tactical  combat  assets.  Their  decisions  determine 
how  effectively  the  available  assets  are  assigned  to  objectives  or  targets. 

Tactical  battle  staff  officers  currently  practice  decision-making  skills  during  manpower-intensive 
Command  Post  Exercises  (CPXs),  which  are  supported  by  computer-based  simulations  running  on 
medium-  and  large-scale  computer  systems.  Participation  in  such  exercises  is  sporadic  and  expensive.  It 
is  therefore  difficult  for  battle  staff  officers  to  achieve  and  maintain  the  skill  levels  required  for  successful 
combat  operations.  As  a  result,  current  training  for  decision-making  skills  is  basically  not  cost-effective 
(Brecke,  Jacobs,  &  Krebs,  1988).  There  is  a  clear  need  to  make  decision  training  more  accessible,  more 
affordable,  and  less  manpower-intensive. 

This  need  has  been  recognized,  and  a  technology  base  that  can  provide  easily  accessible  and 
affordable  training  at  the  unit  level  is  emerging.  Affordability  is  achieved  through  the  use  of  increasingly 
powerful  microprocessor-based  computers.  The  need  for  instructors  and  live  participants  is  reduced 
through  advanced  modeling  techniques  made  possible  by  expert  systems  technology.  Wilson  (1982) 
described  an  “Interactive  Micro-Computer  Wargame  for  an  Air  Battle”  running  on  an  Apple  III  and 
written  in  UCSD  Pascal.  Obermayer,  Johnston,  Slemon,  and  Hicklin  (1984)  developed  a  micro-based 
(WICAT  System  150)  multi-user  system  which  simulated  a  simple  Anti-Submarine  Warfare  (ASW) 
scenario.  The  system  featured  one  simulated,  knowledge-based  team  member.  Madni,  Ahlers,  and  Chu 
(1987)  demonstrated  the  modeling  of  an  intelligent  opponent  in  a  knowledge-based  simulation  prototype 
running  on  a  Symbolics  3670  workstation.  The  prototype  was  designed  to  train  Tactical  Action  Officers  in 
the  kind  of  decision-making  skills  needed  during  naval  surface  warfare.  The  Army  Research  Institute  for 
the  Behavioral  and  Social  Sciences  (ARI)  has  focused  on  experimental  Computer-Aided  Instruction 
(CAI)  in  the  training  of  decision-makihg  skills  for  armor  officers.  One  result  of  this  work  is  the  “Armor 
Tactical  Concepts  Tutor  (ART ACT),”  which  runs  on  the  Army’s  microcomputer-based  Electronic 
Information  Delivery  System  (EIDS).  Stoddard,  Kern,  and  Emerson  (1986)  announced  the  development 
of  a  cognitive  skills  tutor  which  continues  and  expands  the  ART  ACT  work. 
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The  overall  goal  of  the  present  effort  is  to  add  to  this  emerging  technology  base.  More  precisely,  the 
project  is  designed  to  produce  training  and  training  design  technologies  for  interactive  training  experiences 
which  reliably  result  in  the  acquisition  of  decision-making  skills  and  are  presented  by  means  of 
microcomputer  media  without  a  human  instructor.  The  specific  objectives  of  the  project  were  defined  as 
follows: 

Objective  1:  Develop  a  methodology  to  externalize  expert  task  knowledge  for 

decision-making  tasks. 

Objective  2:  Develop  training  methodologies  which  facilitate  achievement  of  reliable  and 

high  performance  of  decision-making  tasks. 

Perspective 

The  project  in  its  entirety  is  viewed  as  an  Instructional  Systems  Design  (ISD)  problem  where  a  generic 
training  design  solution  for  a  class  of  skills  (decision-making)  is  sought.  An  ISD  problem  exists  if  some  of 
the  variables  that  define  an  instructional  event  or  product  are  “given”  and  others  are  “unknown”  or  “to 
be  determined."  Merrill  and  Wood  (1974)  identified  four  “relatively  independent”  facets  of  instruction 
representing  sets  of  instructional  variables:  Learner  Aptitude,  Content  (or  subject  matter),  Instructional 
Strategy,  and  Delivery  System.  Frank  (1969),  in  a  more  formal  cybernetic  theory  of  instruction,  used 
basically  the  same  variable  sets  or  dimensions  and  added  two  more:  Purpose  (or  Objective)  of  Instruction 
and  Environment  within  which  instruction  is  to  take  place. 

The  ISD  problem  to  be  solved  by  this  project  can  thus  be  specified  as  one  where  four  variable  sets  or 
dimensions  (in  Frank’s  terms)  are  given  and  two  are  sought.  The  following  variables  are  given  or 


predetermined: 

1. 

The  Training  Objective 

Specified  as  a  class  of  skills  to  be  trained:  Decision- 
Making  Skills 

2. 

The  Learners 

Battle  Staff  Officers  assigned  to  Tactical  Command 
and  Control  Systems 

3. 

The  Delivery  System 

Microcomputer  media 

4. 

The  Environment 

In-house  environment  at  Tactical  Command  and 
Control  Units 

Solving  the  ISD  problem  involves  defining  the  two  remaining  dimensions  which  are  required  to 
completely  specify  an  instructional  event: 

5.  The  Content  Classes  and  instances  of  domain  knowledge 

6.  The  Instructional  Strategy  Types  of  instructional  actions  and  their  sequencing. 
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Prior  Work  in  Phases  I  and  II 


The  focus  of  the  first  two  phases  of  this  three-phase  project  was  on  the  acquisition  of  instructional 
content  (Objective  1).  Instructional  content  for  decision  training  consists  of  the  task  knowledge  employed 
by  experts  during  the  act  of  making  decisions.  This  task  knowledge  resides  not  in  books,  regulations,  or 
pamphlets 'but  in  the  minds  of  experts.  One  way  to  make  this  knowledge  explicit  is  with  the  aid  of 
knowledge  engineering  techniques.  More  traditional  task  analysis  techniques  have  proven  to  be  ineffective 
for  this  purpose  (Brecke  et  al.,  1988).  Unaided,  interview-type  knowledge  engineering  methods  were 
successful  up  to  a  point  but  encountered  significant  problems. 

A  computer-based  (Smalltalk  and  PC/AT)  Knowledge  Engineering  and  Training  System  (KEATS) 
was  developed  to  aid  in  the  knowledge  acquisition  process  and  to  overcome  the  problems  encountered 
(Brecke  et  al.,  1989).  During  two  evaluation  trials,  KEATS-aided  knowledge  engineering  sessions 
recorded  135  justifications  for  28  decision  instances.  Subsequent  analysis  produced  64  decision  rules. 
These  results  were  interpreted  as  a  proof  of  concept  for  computer-aided  acquisition  of  instructional 
content  for  decision-making  tasks.  Approximately  70%  of  the  KEATS  software  is  generic. 
Domain-specific  KEATS  applications  for  other  domains  can  therefore  be  generated  at  relatively  low  cost. 
The  methodology  represented  by  KEATS  thus  satisfies  the  first  project  objective. 

Phase  III  Problem  and  Objective 

In  Phase  III,  the  focus  of  the  project  was  on  the  second  project  objective:  the  development  of  training 
methodologies  which  facilitate  acquisition  of  decision-making  skills.  Attention  in  this  phase  thus  shifted 
away  from  the  content  variable  of  the  instructional  problem  to  the  instructional  strategy  variable.  Within 
the  overall  context  of  instructional  strategy,  Phase  III  efforts  concentrated  on  the  strategy  components 
practice  and  feedback,  which  are  essential  elements  of  any  instructional  strategy  for  any  type  of  training 
objective.  Specifically,  the  objective  of  Phase  III  was  to  design  and  develop  an  affordable  computer-based 
practice  environment  that  would  promote  the  acquisition  of  specific  decision-making  skills  in  the  target 
domain  of  a  Fighter  Duty  Officer  (FDO)  at  an  Air  Support  Operations  Center  (ASOC). 

Report  Scope 

The  present  report  covers  Phase  III.  It  shows  how  the  requirements  for  the  practice  environment 
were  derived,  how  the  system  (called  SuperKEATS)  was  developed,  and  how  decision-making  practice  in 
SuperKEATS  occurs.  This  description  includes  SuperKEATS'  facilities  to  modulate  practice  and 
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feedback  strategies  to  accommodate  variations  in  learner  abilities  and  needs,  as  well  as  to  permit 
experimental  research.  The  report  also  describes  a  2-day,  two-subject  evaluation  of  SuperKEATS,  and 
ends  with  a  discussion  and  conclusions  regarding  the  viability  of  the  training  methodology  represented  by 
SuperKEATS. 

II.  APPROACH 

This  secuon  describes  the  approach  followed  in  designing  and  developing  a  practice  environment  for 

decision-making  in  a  specific  tactical  decision  domain.  We  show  first  how  specific  requirements  for  the 

practice  environment  were  derived.  The  design,  development  and  implementation  steps  are  described 

next.  This  is  followed  by  a  description  of  the  internal  structure  of  the  system  called  SuperKEATS. 

Finally,  SuperKEATS  is  described  in  terms  of  how  it  is  used,  both  from  the  perspective  of  the  student 

practicing  decision-making  skills  and  from  the  perspective  of  the  researcher  experimenting  with  instructional 
strategies. 


Requirements  for  a  Decision  Practice  Environment 

KEATS  provided  a  means  to  acquire  the  content  that  should  be  communicated  by  a  decision  training 
system.  The  issue  of  how  this  communication  should  be  structured  is  a  matter  of  defining  an  appropriate 
instructional  strategy.  Below  we  examine  the  concept  of  instructional  strategy  in  general,  focus  on  the 
practice  component  which  must  be  present  in  any  instructional  strategy,  and  derive  a  set  of  molecular 
strategy  requirements  which  must  be  satisfied  by  a  practice  environment  for  decision-making  skills. 

The  Concept  of  Instructional  Strategy 

An  instructional  strategy  is  basically  a  set  of  rules  which  prescribe  a  particular  sequence  of 
instructional  elements.  Frank  (1969)  used  the  term  “teaching  algorithm”  to  describe  the  notion  of  an 
instructional  strategy.  Aagard  and  Braby  (1976)  spoke  of  “Learning  Guidelines  and  Algorithms  for 
Twelve  Types  of  Training  Objectives.”  They  presented  flowcharts  depicting  sequences  of  learner  and 
instructional  system  activities  which  have  the  flavor,  albeit  not  the  rigor,  of  formal  algorithms.  Merrill 
(1983)  favored  the  term  “organizational  strategy”  and  defined  it  as  “decisions  involved  in  the  design  of 
learning  activities,  including  the  types  of  displays  to  be  presented  to  the  student,  the  sequence  of  these 
displays,  the  topics  to  be  included,  the  sequence  and  structure  of  these  topics,  the  type  of  practice,  the 
nature  of  feedback,  and  other  decisions  regarding  the  nature  of  the  presentation.” 
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Instructional  strategies  can  be  defined  at  various  levels  of  resolution.  Frank  (1969)  and  Merrill  (1983) 
distinguished  between  macro  and  micro  strategies.  Macro  strategies  define  the  sequence  of  subject-matter 
topics  (Merrill,  1983)  or  objectives  (Brecke,  1976).  Micro  strategies  define  the  sequence  and  form  of 
instructional  events  within  an  instructional  module  for  a  topic  or  objective  (Brecke,  1973).  Other  authors 
spoke  of  molar  and  molecular  levels  of  instructional  strategy  (e.g.,  Gropper,  1974).  In  essence, 
instructional  strategies  can  be  defined  on  a  continuum  of  granularity:  on  the  level  of  a  curriculum  as  a 
sequence  of  courses;  on  the  level  of  a  syllabus  as  a  sequence  of  lessons;  on  the  level  of  a  lesson  as  a 
sequence  of  objectives;  and,  finally,  on  the  level  of  an  objective  as  a  sequence  of  instructional  activities  or 
elements.  Here  the  terms  molar  and  molecular  will  be  employed  to  avoid  confusion  with  the  macro  and 
micro  aspects  of  the  target  training  task  that  will  be  introduced  later. 

On  the  molecular  level,  instructional  strategy  follows  a  basic  and  universal  paradigm  of  three 
successive  phases:  Presentation,  Practice  and  Testing  (Merrill,  1983).  Variations  in  instructional 
strategies  occur  one  level  below  this  paradigm  within  each  of  the  phases.  In  Phase  III  (of  this  project),  we 
focused  on  the  molecular  levels  of  both  instruction  and  the  practice  phase  of  instruction.  As  indicated 
before,  our  goal  was  to  design  a  suitable  practice  environment  for  decision-making  skills.  We  were 
therefore  concerned  with  the  relatively  narrow  problem  of  defining  molecular  instructional  strategies  that 
would  optimize  learning  of  decision-making  skills  during  the  practice  phase  of  instruction. 

Instructional  Strategy  Based  on  Component  Display  Theory 

Prescriptions  for  instructional  strategies  can  be  obtained  from  two  sources:  Instructional  Design 
Theories  and  Task  Analyses.  Instructional  design  theories  and  prescriptive  theories  are  generally  founded 
on  two  types  of  descriptive  theories:  theories  of  learning  or  skill  acquisition,  and  theories  of  skill 
performance.1  The  strategy  prescriptions  which  can  be  derived  from  instructional  design  theories  generally 
address  molar  as  well  as  molecular  levels. 

The  most  detailed  in  tructional  design  theory  that  is  currently  available  for  the  molecular  level  is 
Merrill’s  Component  Display  Theory  or  CDT  (Merrill,  1983).  CDT  is  based  on  Gagne’s  postulate  (1965) 
that  different  types  of  learner  outcomes  require  different  conditions  of  learning  and  on  a  classification 
scheme  for  learner  outcomes  which  is  known  as  the  Performance  Content  Matrix  (Figure  1).  Each  cell  of 
the  matrix  defines  a  different  class  of  learner  outcome  or  type  of  training  objective.  CDT  offers 


’For  an  interesting  discussion  of  the  differences  between  descriptive  and  prescriptive  theories  of 
learning  and  instruction,  see  Reigeluth,  1983,  pp.  21-25. 
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FACT  CONCEPT  PROCEDURE  PRINCIPLE 
TYPES  OF  CONTENT 

Figure  1.  Performance-Content  Matrix  (Merrill,  1983). 


instructional  strategy  prescriptions  for  each  type  of  objective,  where  the  prescriptions  are  formulated  in  a 
“language”  that  includes  four  primary  presentation  forms  and  eight  secondary  presentation  forms. 

Training  objectives  aimed  at  the  acquisition  of  decision-making  skills  can  be  classified  in  terms  of 
Merrill’s  Performance-Content  Matrix,  which  means  that  a  corresponding  molecular  strategy  for 
decision-making  objectives  should  accordingly  be  available.  The  most  appropriate  category  for 
decision-making  objectives  is  the  USE-PRINCIPLE  cell.2  For  the  performance  category  USE,  CDT 
prescribes  (regardless  of  content  category)  Iegs.N  for  the  practice  phase;  i.e.,  the  use  of  a  set  of  two  or 
more  “ Inquisitory ”  examples  (egs)  or  instances.  The  specific  form  of  these  practice  items  is  prescribed  as 
follows: 

The  instance  practice  for  a  principle  entails  the  presentation  of  a  problem  situation  to 
the  student.  The  name  of  the  principle  may  or  may  not  be  given,  and  the  student  is 
asked  to  explain  what  happens.  This  explanation  may  take  the  form  of  a  prediction 
about  the  outcome,  or  it  may  take  the  form  of  a  solution  to  the  problem. 
Representation  [again]  plays  an  important  role.  If  it  is  impossible  to  have  the  actual 
objects  and  events  present,  then  they  must  be  represented.  The  representation  must 
be  sufficient  so  that  all  of  the  critical  aspects  of  the  cause-and-effect  relationship  are 
represented  to  the  student.  (Merrill,  1983,  p.  318) 

2“Use  means  to  use  a  general  rule  to  process  specific  information”  (Merrill,  1983,  p.  303). 

“  Principles  are  those  cause-and-effect  or  correlational  relationships  that  are  used  to  interpret  events 
or  circumstances”  (Ibid.,  p.  288). 
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This  is  an  admirably  detailed  prescription  if  one  considers  the  myriad  of  different  learning  outcomes 
that  might  fall  under  the  USE-PRINCIPLE  class.  It  is,  however,  far  too  imprecise  to  serve  as  a  principled 
design  specification  for  a  part'  ular  subclass  of  skills  such  as  decision-making  skills. 


Cognitive  Task  Analysis 
• 

At  this  point,  the  second  source  of  instructional  strategy  prescriptions.  Task  Analysis,  becomes  useful. 
Decision-making  tasks  require  a  particular  type  of  task  analysis.  The  conventional  methods  of  successive 
task  decomposition  into  smaller  and  smaller  steps  (resulting  in  hierarchical  and/or  flowchart 
representations)  have  been  found  to  be  ineffective  (Brecke  et  al.,  1988)  for  complex  cognitive  (and  thus 
covert)  tasks.  These  types  of  tasks  require  a  method  that  elucidates  the  essential  cognitive  events  that 
occur  during  task  performance.  Lesgold  et  al.  (1986)  called  this  type  of  analysis  “Cognitive  Task 
Analysis”  (see  also  Means  &  Gott,  1988). 

A  cognitive  task  analysis  of  the  targeted  decision-making  skill  emerged  as  a  byproduct  of  the 
knowledge  acquisition  efforts  during  Phase  II  of  the  project.  The  intent  of  the  knowledge  acquisition 
phase  was  to  obtain,  in  explicit  form,  expert  knowledge  employed  during  task  performance.  Such 
knowledge  was  obtained  and,  in  the  course  of  its  acquisition,  a  model  of  the  cognitive  events  involved  in 
the  target  task  emerged.  Thus,  there  was  neither  the  intent  to  perform  a  cognitive  task  analysis  nor  a 
desire  to  follow  Lesgold’s  methods,  yet  the  initial  interviews  naturally  focused  on  the  technical  and 
cognitive  aspects  of  the  target  decision  domain  and  the  specific  decision  task.  The  product  of  these  efforts 
was  a  detailed  understanding  of  the  target  domain  and  a  cognitive  model  of  the  target  decision  task.  This 
product  is  briefly  outlined  below. 

The  Study  Domain.  The  general  target  domain  for  the  project  is  the  Tactical  Command  and  Control 
System  of  the  Air  Force.  Within  the  general  domain  of  Air  Force  Tactical  Command  and  Control,  the 
focus  is  on  a  specific  type  of  Tactical  Command  and  Control  node  called  the  Air  Support  Operations 
Center  (ASOC).  The  ASOC  is  an  Air  Force  Tactical  Command  and  Control  unit  which  is  “assigned”  to 
and  co-located  with  an  Army  Corps  Headquarters  (see  Figure  2).  Its  basic  role  is  to  supply  air  support 
missions  in  response  to  demands  originating  from  the  Army  Corps’  front-line  divisions.  The  ASOC  thus 
has  to  solve  a  problem  which,  in  general  terms,  can  be  viewed  as  a  supply  management  problem.  The 
problem  is  nontrivial  because  the  demand  for  air  support  may  exceed  the  supply  of  available  aircraft,  and 
both  supply  and  demand  vary  independently  over  time  in  ways  that  are  only  partially  predictable. 
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Figure  2.  Project  Study  Domain:  Air  Support  Operations  Center  (ASOC). 

The  ASOC’s  method  for  solving  this  supply  management  problem  is  a  combination  of  advance 
planning  and  subsequent  plan  execution  with  ad-hoc  adjustments.  Advance  planning  is  done  during  the 
night  shift  when  little  or  no  combat  action  is  occurring.  Plan  execution,  often  referred  to  as  “operations,” 
occurs  during  the  daylight  hours  when  most  tactical  combat  action  occurs. 

The  plan  produced  by  the  night  shift  crew  is  basically  a  schedule  that  maps  predicted  supplies  of 
available  sorties  into  anticipated  demands  for  air  support  for  each  of  the  corps’  front-line  divisions.  This 
plan  or  schedule  covers  the  daylight  hours;  i.e.,  the  time  from  first  light  to  last  light.  Information  for  the 
supply  side  of  the  plan  comes  pnmarily  from  the  Air  Force  Air  Tasking  Order  (ATO)  and  Operations 
Order  (OO) .  Information  for  the  demand  side  of  the  plan  comes  primarily  from  the  Army  Corps'  plans 
and  from  intelligence  sources.  The  Army  has  the  prerogative  of  determining  the  "priority  of  fire,”  which 
is  defined  in  percentage  allocations  of  the  total  available  sorties  to  each  of  the  front-line  divisions.  The 
ASOC  determines  the  details  of  time  frames,  squadron  assignments,  and  tasking  modes. 


Plan  execution  is  essentially  the  responsibility  of  the  day  shift  crew.  The  day  crew  uses  the  plan  as  a 
baseline  which  usually  has  to  be  modified  in  view  of  the  most  recent  information  regarding  the  available 
assets  and  the  battlefield  situation.  The  plan  is  adjusted  and  readjusted  throughout  the  day  in  response  to 
changes  in  the  tactical  situation  and  as  preparation  for  anticipated  developments  in  the  near  future.  The 
key  to  successful  air  support  service  from  the  ASOC  to  the  corps  is  to  manage  sortie  generation  and  sortie 
expenditure  such  that  the  corps  receives  optimal  support  in  the  present,  throughout  the  day,  and  over  the 
foreseeable  length  of  a  conflict. 
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Our  focus  was  on  the  decision  problems  of  the  day  crew;  i.e.,  on  the  “OPERATIONS”  side  of  the 
ASOC.  The  day  crew  deals  both  with  the  present  and  with  the  near  future,  and  with  actual  and 
anticipated  supply  and  demand  problems.  The  decision  problems  faced  by  the  day  crew  thus  occur  on 
two  levels:  on  the  level  of  specific  and  immediate  decisions  regarding  particular  requests,  and  on  the  level 
of  an  overall  plan  that  takes  into  account  the  “big  picture”  over  a  longer  time  horizon.  We  refer  to  these 
two  levels  as  the  micro  and  macro  levels  of  decision-making  at  the  ASOC.  The  two  levels  are 
interdependent:  Micro-level  decisions  are  made  under  the  constraints  of  macro  guidance,  and  micro 
events  lead  to  adjustment  decisions  on  the  macro  or  plan  level. 

The  ASOC  operations  team  consists  of  six  officers  and  three  noncommissioned  officers  (NCOs)  called 
“technicians.”  The  NCOs  share  in  the  officers’  workload  of  decision-making  tasks  and  implement  the 
officers’  orders  and  provide  other  assistance  as  needed.  The  Director  oversees  the  entire  ASOC 
operation.  He  is  primarily  concerned  with  overall  situation  assessment  and  advance  planning,  and  issues 
guidelines  to  the  rest  of  the  team.  The  Fighter  Duty  Officer  1  (and  Technician),  referred  to  as  the  FDO 
(FIDO),  assists  the  Director  in  situation  assessment  and  planning  and  assigns  air  resources  to  air  requests. 
He  also  coordinates  mission  support  requirements  between  Army  and  Air  Force  in  conjunction  with  the 
G-3  Air.  The  Fighter  Duty  Officer  II  (and  Technician)  is  essentially  an  apprentice  FDO.  In  addition, 
there  are  a  Reconnaissance  Duty  Officer,  a  G-3  Air,  and  an  Intelligence  Officer  (and  Technician). 

The  following  examples  illustrate  the  types  of  decision-making  skills  that  need  to  be  trained. 
Example  1: 

The  ASOC  Fighter  Duty  Officer  1  (FDO  I)  receives  an  air  request  to  neutralize  eight  tanks  within  the 
next  hour.  The  request  has  been  verified  by  the  appropriate  officers.  The  FDO  I  must  now  decide 
which  of  the  available  aircraft  assets  should  be  tasked  to  satisfy  this  request.  If  he  has  no  assets  to 
task,  he  must  refuse  the  request.  If  weather  or  threats  make  mission  success  questionable,  he  has  to 
weigh  risks  against  benefits. 

Example  2: 

The  ASOC  receives  a  Corps  request  to  neutralize  a  large  number  of  tanks  attacking  along  a  highway. 
This  type  of  request  generally  requires  tasking  of  a  fairly  large  number  of  aircraft,  some  of  which  will 
be  the  actual  attackers  and  some  of  which  will  fly  support  missions.  This  type  of  request  may  include 
participating  Army  assets.  This  sort  of  decision  problem  is  usually  handled  by  the  ASOC  Director, 
who  decides  on  force  composition,  timing,  coordination,  and  other  relevant  factors. 

In  either  of  the  examples  above,  the  decision-maker  has  to  consider  multiple  factors.  Some  of  the 
most  important  factors  are  depicted  in  Figure  3. 
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Figure  3.  Factors  Considered  by  Tactical  Command  and  Control  Decision  Makers. 

The  Target  Training  Objective.  In  order  to  keep  the  effort  within  manageable  limits,  the  prototype 
system  had  to  be  aimed  at  a  single  and  specific  target  training  objective.  This  objective  had  to  be 
representative  of  the  decision  skills  exercised  on  the  offensive  side  of  tactical  command  and  control  in  the 
Air  Force.  In  earlier  work  (Brecke  et  al.,  1988),  task  analyses  were  performed  for  a  number  of  positions 
at  the  Allied  Tactical  Operations  Center  (ATOC).  These  analyses  and  the  work  performed  in  Phase  II  of 
this  project  (Brecke  et  al.,  1989)  showed  that  the  ASOC  FDO  performs  a  task  which  is  nearly  identical  to 
several  tasks  occurring  in  the  ATOC.  This  task  was  used  as  a  basis  for  specifying  the  following  specific 
training  objective  as  the  target  objective  for  the  training  system  prototype: 

Decide  on  the  disposition  of  air  requests  over  an  entire  day  operations  shift,  given  a 
simulated  tactical  situation  for  an  ASOC  and  the  Army  Corps  it  supports. 

This  objective  is  valid  not  only  for  the  FDO  but  for  all  officers  in  the  ASOC  van.  It  represents  the 
essence  of  what  the  ASOC  does  and  it  is  an  activity  to  which  all  must  contribute  in  a  coordinated  fashion. 
The  ASOC  van  team  must,  in  fact,  have  a  shared  mental  model  as  to  how  to  do  this  task.  The  objective  is 
therefore  generic  as  far  as  the  ASOC  is  concerned.  The  objective  also  generalizes  easily  to  decision  tasks 
performed  in  the  ATOC. 
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The  objective  does  not  state  a  performance  standard.  The  measurement  and  evaluation  of 
decision-making  performance  was  left  open  as  a  research  issue  that  should  be  investigated  with  the 
prototype  training  system. 

Cognitive  Model  of  the  Target  Task.  A  cognitive  model  of  the  task  represented  by  the  objective 
above  is  presented  in  Figure  4  and  the  description  that  follows. 

The  FDO’s  decision-making  task  is  initiated  by  Eliciting  Stimuli  which  consist  of  messages  passed  to 
the  ASOC  via  a  number  of  communication  links.  These  messages  consist  of  air  requests  (i.e.,  requests  for 
air  support  from  units  of  the  Army  Corps  supported  by  the  ASOC)  and  other  messages  which  inform  the 
ASOC  of  changes  in  the  availability  of  air  assets.  The  messages  also  include  feedback  on  the  results  of 
previously  tasked  missions.  We  focus  here  on  air  request  messages. 

These  messages  elicit  a  Decision  Process,  which  is  roughly  subdivided  into  four  stages.  During  the 
first  stage,  a  verification  and  priority  filtering  process  takes  place  in  which  other  members  of  the  ASOC 
team  participate.  The  request  must  be  verified  as  legitimate  and  its  priority  in  terms  of  time  and 
competing  requests  must  be  determined.  The  request  may  at  this  point  be  refused  or  further  clarification 
may  be  requested.  If  the  filter  is  passed  successfully,  the  FDO  locates  and  selects  suitable  primary  assets 
(aircraft).  The  FDO  then  determines  whether  support  assets  (e.g.,  electronic  warfare  and  other  defense 
suppression  assets  from  either  the  Army  or  the  Air  Force)  are  required  and  available.  Finally, 
coordination  details  regarding  timing,  contact  points,  and  radio  frequencies  are  determined.  The  request 
may  get  refused  anywhere  along  this  process,  depending  on  asset  availability  and  mission  priority. 

During  this  entire  process,  the  FDO  must  have  access  to  an  information  environment  which  must 
contain  at  least  the  following  major  types  of  information:  Air  Asset  Status  (available  from  status  boards 
and  directly  from  Wing  Operations  Centers  or  WOCs);  Air  Force  Orders  (Air  Tasking  Order  or  ATO, 
Daily  Operations  Order  or  DOO);  Army  Corps  Guidance  (plans  and  fire  priorities);  Intelligence  (on 
enemy  plans  and  movements);  and,  finally,  an  image  of  the  tactical  situation  represented  by  means  of 
maps. 

Also,  along  the  way  through  this  process  the  FDO  must  be  aware  of  the  current  operations  plan.  His 
decisions  regarding  particular  air  requests  (i.e.,  his  decisions  on  the  micro  level)  should  be  guided  by  the 
current  overall  plan,  which  is  a  reflection  of  macro  decisions.  He  must  constantly  reassess  this  plan  for  its 
continued  viability  in  the  face  of  new  incoming  information  and  decide  whether  it  is  time  to  change  the 
plan  and  how  it  should  be  changed.  This  macro  decision  process  goes  on  in  the  background  while  the 
FDO  deals  with  the  micro  decisions  required  to  satisfy  particular  requests. 
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Figure  4.  Cognitive  Model  of  the  Target  Decision  Task. 


The  output  of  the  decision  process  is  an  Implementing  Response,  which  consists  of  various  types  of  air 
tasking  messages  that  are  sent  via  communication  nets  to  the  operational  units  (WOCs  or  squadrons)  that 
own  the  airplanes  which  will  fly  the  air  support  missions. 

The  implementing  response  thus  triggers  an  operational  response  which  has  some  effect  on  the 
Tactical  Environment ;  i.e.,  on  the  course  of  the  ground  battle.  If  airplanes  are  lost  during  missions,  the 
air  asset  status  is  affected  as  well. 

Feedback  on  the  effects  of  the  operational  responses  reaches  the  FDO  by  way  of  mission  reports, 
which  are  issued  either  while  missions  are  still  airborne  or  after  their  return  to  their  base  of  origin  or  to  an 
alternate  base.  Further  feedback  reaches  the  FDO  via  intelligence  reports  and  via  changing  plans  and  fire 
priorities  from  the  Army  Corps. 

A  particularly  important  source  of  feedback  is  the  changing  pattern  of  subsequent  air  requests.  For 
example,  an  area  may  previously  have  produced  very  frequent  requests  on  targets  that  seemed  to  be 
moving  more  and  more  into  friendly  territory.  The  FDO  has  responded  to  these  demands  with  high  levels 
of  air  support.  If,  as  a  consequence,  the  frequency  of  requests  decreases  and  if  the  targets  appear  to  be 
moving  out  of  friendly  and  into  hostile  territory,  then  the  FDO  can  legitimately  deduce  that  his  decisions 
were  effective. 

Functional  Specifications  for  the  Prototype  Decision  Practice  System 

The  cognitive  model  described  above  furnished  a  level  of  detail  beyond  Merrill’s  fairly  global 
prescriptions  for  USE-PRINCIPLE  objectives.  In  essence,  it  provides  the  rationale  for  constructing  a 
representation  of  decision  problems  that  is  “sufficient  so  that  all  of  the  critical  aspects  of  the 
cause-and-effect  relationship  are  represented  to  the  student”  (Merrill,  1983,  p.  318).  It  allows  the 
formulation  of  specific  functional  requirements  that  a  system  for  decision-making  practice  should  satisfy. 

We  make  two  important  assumptions  at  this  point.  The  first  assumption  is  that  a  practice  environment 
which  replicates  the  ASOC  environment  with  all  its  essential  objects  in  a  functional  rather  than  literal 
sense,  and  which  allows  the  student  to  exercise  the  cognitive  model  described  above,  will  have  a  positive 
effect  on  the  acquisition  of  the  target  decision-making  skill  and  that  there  will  be  positive  transfer  from 
the  skill  so  acquired  in  the  practice  environment  to  performance  in  the  real-world  operational 
environment. 
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The  second  assumption  is  that  practice  is  most  effective  if  it  is  modulated  in  complexity  and  supported 
by  instructional  features  like  prompts  (which  are  secondary  presentation  forms  in  Merrill's  CDT)  and 
artificial  instructional  feedback.  Such  modulations  and  support  should  occur  on  the  basis  of  changes  in 
student  skill  level. 

Both  of  these  assumptions  need  to  be  tested  empirically.  The  prototype  practice  environment  must 
therefore  functionally  replicate  the  ASOC  environment  and  permit  experimentation  with  different 
complexity  schedules  and  levels  of  instructional  support. 

Together  these  assumptions  and  the  cognitive  model  led  directly  to  the  functional  specifications 
presented  in  Table  i.  The  functional  specifications  had  to  be  tempered  against  those  development 
constraints  described  in  the  next  section. 

Development  Constraints 

The  primary  resource  constraints  were  time,  funding,  and  the  hardware/software  selected  for 
implementation.  The  calendar  time  available  from  design  start  to  evaluation  was  12  months.  The 
software  to  be  used,  Knowledge  Engineering  Environment  (KEE),  had  been  selected  and  purchased 
during  Phase  I  of  the  project.  The  hardware  (Table  2),  two  Sun  3/60  workstations,  was  purchased,  in 
place,  and  configured  to  run  KEE  prior  to  starting  the  design. 

To  build,  under  these  constraints,  a  system  that  meets  the  above  goals  meant  that  certain  limitations 
on  the  capabilities  of  the  simulation  environment  had  to  be  accepted.  These  limitations,  given  in  Table  3, 
were  chosen  to  simplify  the  simulation  modeling  while  retaining  all  essential  FDO  decision-making 
parameters. 

Development  of  the  SuperKEATS  Practice  Environment 

Development  of  SuperKEATS  initially  concentrated  on  the  simulation  portion  of  the  system.  After  a 
rudimentary  simulation  was  implemented  such  that  a  user  could  “play  the  ASOC  game,”  instructional 
features  were  added.  The  development  followed  a  straightforward  sequence  of  design, 
implement/integrate,  and  test.  These  development  steps  are  described  briefly  below. 

The  design  featured  an  object-oriented  approach  much  like  that  of  the  precursor  KEATS  system. 
Objects  were  grouped  as  simulation,  Army,  Air  Force,  interface,  event,  and  other  objects.  Associated 
with  each  object  were  its  attributes,  input  messages,  a  brief  description  of  the  processing  of  each  input, 
and  output  messages.  Attributes  define  the  state  of  an  object.  In  an  object-oriented  paradigm,  all  objects 
communicate  with  each  other  via  messages.  Hence,  the  input  messages  define  what  an  object  must 
respond  to  and  the  output  messages  define  what  the  object  will  send  or  ask  of  other  objects. 
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Table  1 .  Functional  Specifications  for  the  Prototype  Decision  Practice  Environment  (SuperKEATS) 


ELEMENTS 

FUNCTIONAL  SPECIFICATIONS  j 

MACRO-LEVEL 

MICRO-LEVEL 

EucrnNG 

STIMULI 

PROVIDE  MESSAGES  THAT  ELICIT 

CHANGES  TO  SITUATION  ASSESSMENT 

AND  PLANNING 

PROVIDE  MESSAGES  THAT  ELICIT 

DECISIONS  ON  AIR  REQUEST 

DISPOSITION 

DECISION 

PROCESS 

PROVIDE  ACCESS  TO  CURRENT 

OPERATING  PLAN  FOR  REVIEW 

PROVIDE  ACCESS  TO  CURRENT 

OPERATING  PLAN  FOR  REVIEW 

PROVIDE  ACCESS  TO  AN  INFORMATION 
ENVIRONMENT  CONTAINING  AT  A 

MINIMUM 

PROVIDE  ACCESS  TO  AN  INFORMATION 
ENVIRONMENT  CONTAINING  AT  A 

MBNMUM 

•  Affl  ASSET  STATUS 

•  AIR  FORCE  ORDERS 

•  ARMY  CORPS  GUIDANCE 

•  INTELLIGENCE 

•  MAP 

•  Am  ASSET  STATUS 

•  ARMY  CORPS  GUIDANCE 

•  MAP 

PROVIDE  CAPABILITY  TO  CHANGE 

CURRENT  OPERATIONS  PLAN 

PROVIDE  CAPABILITY  TO: 

•  CHANGE  AIR  STATUS 

•  TASK  MISSIONS 

•  REFUSE  MISSIONS 

•  REQUEST  SUPPORT  ASSETS 

FEEDBACK 

(NATURAL) 

PROVIDE  A  SIMULATION  OF  THE 

TACTICAL  ENVIRONMENT  WHICH 

SHOWS  PLAUSIBLE  EFFECTS  OF  AIR 
SUPPORT 

PROVIDE  A  SIMULATION  OF 

AIR  FORCE  OPERATIONAL  UNITS 

WHICH  GIVE  PLAUSIBLE  RESPONSES 
TOTASKINGS 

FEEDBACK 

(ARTIFICIAL) 

PROVIDE  "BEAN  COUNTING"  SCORES  ON 
OVERALL  AIR  SUPPORT  EFFECTIVENESS 

PROVIDE  FOR  TRIAL  FEEDBACK 

PRIOR  TO  DECISION  NPLEMENT ATION 

PROVIDE  MAP  WITH  AUTOMATIC 

DISPLAYS  OF  TARGETS 

PROVIDE  CRITIQUE  OF  TASKMG 

WITH  EXPLANATIONS  AFTER  DECISION 
MPLEMENTATION 

PROVIDE  CAPABILITY  TO  TURN  THE 

ABOVE  TYPES  OF  FEEDBACK  ON  AND 

OFF 

PROMPTS 

PROVIDE  HINTS  FOR  PLAN  CHANGES 

PROVIDE  SUGGESTIONS  FOR  TYPES  OF 
AIRCRAFT  TO  USE  WITH  RATIONALES 

PROVIDE  PLAN  OPTIONS 

PROVIDE  SPECIFIC  OPTIONS  FOR  TASKMG 

PROVIDE  MEANS  TO  TURN  THE  ABOVE 

PROMPTS  ON  AND  OFF 

(THIS  CELL  NOT  IMPLEMENTED 0 

PROVIDE  FOR  CAPABILITY  TO  TURN  THE 
ABOVE  PROMPTS  ON  AND  OFF 

PROVIDE  AN  EXERCISE  BUILDING  CAPACITY  WHICH  PERMITS  GENERATION  OF  MORE  OR 

LESS  COMPLEX  PRACTICE  GAMES  OR  OF  GAMES  WHICH  FOCUS  ON  PARTICULAR  VARIABLES 
(AIRPLANE  TYPE,  BASE  CLOSURE,  TASKING  MODE,  ETC. ) 

PM 

PROVIDE  CAPABILITY  TO  PERMIT  STUDENTS  TO  CONFIGURE  SCREEN  TO  THEM  NEEDS 

H 

PROVIDE  TO  OPERATIONAL  INFORMATION  DISPLAYS  AN  INTERFACE  THAT  IS  EASY  TO 

LEARN  AND  USE,  ELBMNATES  TEDIOUS  BOOKKEEPING  TASKS  AND  RETAINS  FUNCTIONAL 
RATHER  THAN  LITERAL  FIOELITY  TO  OPERATIONAL  INFORMATION  DISPLAYS 
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Table  2.  SuperKEATS  Development  Environment 


SuperKEATS 

0 

0 

o 

Knowledge  Bases 

LISP  Function  Files 

Application  Image  (25  MB) 

0 

KEE  ™  3.1.104 

Programming  Environment 

0 

Sun  Common  LISP  2.1.3 

0 

Unix  3.5  (100  MB  SWAP) 

0 

Sun  3/60  Workstation 

0 

24-MB  RAM 

Hardware  Platform 

0 

327-MB  Hard  Drive 

0 

1152  x  900  Color  Monitor 

o 

HP  Laser  Jet  Plus  Printer 

Table  3.  Simulation  Limitations 

o  Terrain  is  represented  by  a  featureless  plane.  Spatial  considerations  include  position  relative  to 
and  distance  from  the  Forward  Line  of  Own  Troops  (FLOT). 

o  There  is  no  weather.  (Base  attacks  may,  however,  be  used  to  close  bases.) 

o  There  are  no  complex  strategies  for  Army  units  (i.e.,  no  autonomous  Army  units). 

o  There  are  no  complex  Army  unit  movements  (i.e.,  units  move  in  tracks). 

o  There  are  no  front-line  Army  unit  interactions  between  units  in  the  same  Corps  (i.e.,  tracks  do 

not  intersect). 

o  The  front-line  units  are  always  in  “contact"  with  their  opposing  units, 
o  The  only  type  of  Army  units  are  tank  units, 
o  The  only  air  request  targets  are  tanks. 

o  The  ASOC  has  direct  tasking  authority,  with  no  interference  or  help  from  higher  levels  in  the 
Air  Force  command  and  control  hierarchy. 

o  The  WOC  does  not  optimize  the  allocation  of  aircraft  to  missions. 
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In  certain  cases  the  processing  for  a  particular  input  message  was  more  extensive  than  could  be  briefly 
described  in  a  sentence  or  two.  These  cases  were  called  algorithms  and  were  written  up  in  the  detail 
necessary  to  permit  implementation.  In  addition,  because  SuperKEATS  would  be  developed  using  the 
KEE  software  shell,  technical  notes  were  written  about  how  required  interface  features  could  be  most 
efficiently  implemented  in  this  software. 

Upon  completion  of  the  design  phase,  implementation  started  with  a  brief  period  of  prototyping. 
During  this  phase,  the  simulation  objects  were  entered  into  the  KEE  shell  and  typical  displays  with  very 
limited  functionality  we.  a  produced  for  review.  Prototyping  provided  the  framework  for  later 
development,  and  allowed  a  feasibility  and  usability  check  of  interface  concepts. 

Prototyping  was  followed  by  detailed  implementation.  First,  the  models  for  the  Army  and  Air  Force 
were  developed  in  parallel  with  the  interface  for  running  a  game.  Then  functionality  was  added  to  allow 
building  a  game  and  running  a  game  in  real  time.  Also  at  this  point,  the  instructional  features  for 
suggestions  and  feedback  were  introduced  based  on  the  rules  obtained  through  knowledge  acquisition 
from  ASOC  experts  using  KEATS.  Added  late  in  the  development  was  the  display  sequence  for  game 
selection,  assessment,  planning,  and  setup  before  starting  the  game. 

Testing  occurred  at  intervals  during  the  later  stages  of  development  by  having  a  person  who  was  not 
implementing  SuperKEATS  but  was  familiar  with  ASOC  actions  run  the  latest  version,  list  bugs  that  were 
found,  and  recommend  improvements.  Another  test  was  conducted  by  having  a  user-interface  expert  try 
the  system  and  recommend  improvements.  SuperKEATS  was  first  shown  to  members  of  AFHRL/LRG 
for  their  review.  Improvements,  upgrades,  and  additions  were  then  made  before  the  ASOC  members 
used  the  system  and  made  their  evaluations. 

Internal  Structure  of  SuperKEATS 

The  software,  based  on  KEE’s  object-oriented  programming  environment,  is  organized  into  files 
called  knowledge  bases.  Each  knowledge  base  contains  a  group  of  objects,  called  units  in  KEE. 
Functionally  equivalent  to  an  object,  frame  or  schema  in  other  object-oriented  programming 
environments,  a  KEE  unit  represents  the  definable  attributes  and  behaviors  of  a  physical  or  conceptual 
object.  Appendix  A  lists  the  SuperKEATS  knowledge  bases  and  major  units. 
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SuperKEATS’  internal  structure  is  most  easily  described  by  grouping  the  software  into  four 


functionally  related  areas  (each  of  which  comprises  a  set  of  knowledge  bases): 

1.  Simple  Simulation  (SIMSIM):  Simulates  ground  and  air  forces. 

2.  User  Interface :  Displays  textual  and  graphical  information  in  windows,  and  responds  to 
mouse-button  and  menu  selection. 

3.  Instructional  Features:  Allow  user  control  over  gaming  level  and  mode,  artificial  feedback, 
suggestions  and  options,  and  gaming  measures. 

4.  Support  Tools:  Aid  game  building  and  SuperKEATS  system  loading. 

SIMSIM 

SIMSIM  is  an  object-oriented,  agenda-based  simulation  of  air  and  ground  forces.  Included  are 
Army  objects  from  the  corps  level  down  to  the  company  level,  and  Air  Force  objects  such  as  WOCs, 
squadrons,  and  aircraft.  During  game  execution,  the  objects  send  messages  to  each  other  and  the  User 
Interface  to  trigger  ground  and  air  forces  activity.  Simulation  events  placed  on  an  agenda  trigger 
time-delayed  or  scheduled  activity.  The  modeling  and  agenda  processing  are  described  in  more  detail 
below. 

Army  Models.  The  ground  forces  are  modeled  as  two  opposing  Army  Corps  —  a  Red  Corps  and  a 
Blue  Corps  —  with  reserve  and  front-line  tank  units  at  the  division  level  down  to  the  company  level..  All 
front-line  companies  and  their  opposing  companies  move  together  in  close  contact  along  straightline 
paths.  Their  rate  of  movement  depends  on  the  objectives  of  opposing  companies  and  relative  strength 
ratios. 

Strength  ratios  change  as  the  companies  incur  battlefield  losses  and  receive  reserve  reinforcements. 
Ground  battle  attrition  depends  on  the  division  objectives,  strength  ratios,  and  the  actual  number  of  tanks. 
Losses  due  to  close  air  support  depend  on  the  number  of  aircraft,  the  aircraft  type,  and  the  type  of 
ordnance  used. 

Army  objects  are  updated  periodically  at  game-builder-defined  simulation  update  intervals  (typically 
15  or  30  minutes).  During  each  update,  front-line  companies  evaluate  their  resources.  If  needed,  they 
generate  requests  for  reserves  and  air  support.  The  arrival  of  reinforcements  and  the  arrival  of  requests 
for  support  are  both  scheduled  as  agenda  events.  Base  attacks,  duration  of  base  closures  due  to  air  or 
ground  attacks,  and  ground-based  aircraft  losses  are  also  determined  periodically  at  simulation  update 
time. 
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Other  Army  actions  respond  to  aperiodic  events.  Mission  aircraft  losses  and  tank  losses  resulting  from 
missions  are  calculated  when  mission  aircraft  arrive  over  their  targets.  Division  Air  Liaison  Officer  (ALO) 
objects  respond  to  air  request  refusal  and  requests  for  Suppression  of  Enemy  Air  Defenses  (SEAD) 
coordination. 

The  modeling  of  most  Army  actions  required  complex  probabilistic  algorithms. 

Air  Force  Model.  The  Air  Force  model  has  Blue  (friendly)  Air  Force  objects  which  work  closely 
with  the  ASOC.  Because  the  ASOC  is  assumed  to  have  direct  tasking  authority  with  no  interference  or 
help  from  levels  higher  up  in  the  command  and  control  hierarchy,  the  objects  include  WOCs,  squadrons, 
missions,  aircraft  and  bases.  The  user  communicates  directly  with  a  WOC  via  the  User  Interface. 

WOCs  and  squadron  objects  manage  and  account  for  air  assets  as  directed  by  the  user.  WOCs 
respond  to  user-generated  messages  for: 

1.  Tasking,  diverting,  and  canceling  Close  Air  Support  (CAS)  missions. 

2.  Scheduling,  diverting,  or  canceling  a  flow  of  missions  to  a  division  area. 

3.  Establishing,  changing,  and  canceling  aircraft  alerts. 

A  WOC  either  refuses  or  accepts  a  message.  A  WOC  refuses  a  message  if  the  message  violates  any  of 
the  aircraft  resource  constraints  associated  with  that  message.  Refused  messages  have  no  effect  on  the  Air 
Force.  Accepted  messages  result  in  agenda  event  creation  and/or  accounting  updates.  When  a  WOC 
accepts  a  tasking  message,  mission-related  agenda  events  are  created  and  the  mission  is  posted  to  the 
schedule  of  each  mission-assigned  aircraft. 

Additional  messages  sent  by  the  Army  trigger  the  accounting  for  aircraft  lost  during  base  attacks  and 
the  inability  of  missions  to  take  off  or  land  while  a  base  is  closed. 

The  Air  Force  algorithms  are  deterministic,  involving  no  probabilistic  effects. 

Agenda  Event  Processing.  There  are  18  types  of  simulation  events  (Appendix  A’s  Agenda 
Knowledge  Bases).  SIMSIM  objects  create  and  place  events  on  the  agenda  by  sending  messages  to  an 
agenda  manager  object.  The  agenda  manager  maintains  the  events  as  a  time-sorted  list.  When  the 
simulation  time  matches  the  time  of  the  first  listed  event,  the  agenda  manager  removes  the  event  and 
sends  the  event  an  execution  message. 

The  simulation  runs  in  several  modes:  a  user-paced  mode  and  three  real-time  modes.  In  the 
user-paced  mode,  the  user  forwards  the  simulation  to  the  next  event  time.  In  the  real-time  modes,  a 
system  timer  increments  the  simulation  time  at  rates  of  1,  2,  or  10  times  real  time. 


19 


User  Interface 


The  User  Interface  facilitates  user  interactions  with  SIMSIM  and  Instructional  Features.  SIMSIM 
information  is  displayed  in  a  tactical  situation  map  and  as  text  in  user-controllable  windows.  The  user 
communicates  with  SIMSIM  primarily  via  message  forms.  Textual  and  graphical  (graphs  and  plots) 
displays  and  mouse-selectable  menus  support  Instructional  Features. 

SIMSIM  Information  Displays.  As  shown  starting  from  the  top  of  Figure  5,  SIMSIM  generates  air 
requests,  which  are  eventually  displayed  in  a  message  log,  in  an  air  request  display,  and  on  a  map.  A  brief 
summary  of  the  air  request  is  logged  in  the  message  log,  as  are  all  messages  received  by  the  ASOC.  A 
more  detailed  version  of  the  request  is  displayed,  if  the  user  so  chooses,  in  a  separate  air  request  display. 
The  air  request  display  logs  all  missions  tasked  for  the  air  request,  as  well  as  mission  results.  The  map 
displays  the  position  of  the  ground  force  unit  which  generated  the  air  request,  as  well  as  corps,  division, 
boundary  line,  and  Air  Force  base  positions. 

Other  displays  include  information  summaries,  briefings,  orders,  and  status  boards.  Up-to-date 
summaries  for  corps,  divisions,  and  bases,  and  factual  summaries  on  aircraft  and  ordnance  performance, 
are  generated  as  textual  displays  upon  user  request  A  FDO  shift  change  briefing,  a  G-3  Air  briefing,  an 
Intelligence  briefing,  and  an  Air  Tasking  Order  reflect  the  state  of  the  tactical  environment  at  the  start  of 
the  simulation.  Status  boards  show  base,  squadron,  and  mission  status  much  like  the  operational  status 
boards  in  the  ASOC.  These  boards  are  continually  updated  by  SuperKEATS. 

The  user  can  perform  a  variety  of  information  display  window  management  actions,  such  as  open, 
close,  shrink,  expand,  reshape,  and  move  individual  display  windows. 

Transmission  of  User  Communications  to  SIMSIM.  Figure  6  traces  the  message  chain  triggered  by 
user  input  in  response  to  an  air  request  displayed  in  the  message  log.  The  user  fills  in  a  message  form  to 
task  missions  via  mouse  selections  from  menus,  and  keyboard  text  entries.  SIMSIM  responds  to  the 
completed  message  form  by  generating  an  appropriate  response  for  display  in  the  message  log  and,  if 
needed,  updates  status  boards  and  schedules  SIMSIM  events  for  future  processing. 

User  Interface  for  Instructional  Features.  Information  displays  explain  SuperKEATS  use  and  give 
suggestions,  options,  and  game  summaries.  The  user  controls  these  features  by  selecting  buttons  and 
choosing  options  from  pop-up  menus  using  a  mouse.  The  user  fills  in  forms  to  select  a  game,  and  to 
record  an  assessment  of  the  tactical  situation  and  a  plan  of  action. 
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Figure  6.  SIMSIM:  Tracing  the  Processing  of  an  Air  Request. 
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Instructional  Features 


Instructional  features  include  game  control,  suggestions,  options,  and  artificial  feedback  generation. 

Game  Control.  Game  control  prompts  a  user  to  select  a  game  and  then  prepares  SIMSIM  to  run  the 
selected  game.  Game  control  then  guides  a  user  through  the  initial  gaming  setup  steps,  prompting  and 
disabling/enabling  User  Interface  options  appropriate  for  each  gaming  step.  Just  prior  to  and  during  the 
operations  phase  of  the  game,  game  control  supports  user  selection  and  deselection  of  instructional 
features  and  the  setting  of  user-driven  or  real-time  playing  modes.  Game  control  allows  the  user  to  end 
the  game  or  quit  SuperKEATS  at  any  time. 

Suggestions.  Suggestions  are  generated  by  applying  some  of  the  rules-of-thumb  extracted  from 
KEATS  knowledge  acquisition  activities.  The  rules  are  expressed  as  IF-THEN  statements.  IF  conditions 
reference  SIMSIM  object  attributes  such  as  the  number  of  tanks  targeted  in  an  air  request;  THEN  actions 
may  change  the  value  of  bookkeeping  object  attributes  or  generate  a  suggestion  (e.g.,  a  sample 
paraphrased  rule:  “IF  the  number  of  tanks  is  greater  than  20,  THEN  task  more  than  1  mission.”).  If  the 
suggestions  feature  is  selected,  the  suggestion  rules  are  applied  each  time  the  ASOC  receives  an  air 
request.  The  results  are  displayed  in  a  tasking  suggestions  and  options  window. 

Options.  Mission  tasking  options  are  also  produced  and  displayed  for  each  air  request  if  the 
suggestions  feature  is  selected.  Both  suggestions  and  options  are  generated  by  the  Air  Liaison  Officer 
(ALO)  who  receives  the  air  request,  before  passing  the  request  on  to  the  ASOC.  The  ALO  queries 
selected  WOCs  and  squadrons  (in  accordance  with  the  user’s  plan)  to  find  available  missions  to  meet  air 
request  requirements. 

Artificial  Feedback.  There  are  two  types  of  artificial  feedback:  mission-tasking-specific  feedback 
and  game  performance  summary  measures.  Mission  tasking  feedback  is  generated  by  comparing  the 
ALO-suggested  tasking  options  with  user-entered  taskings.  If  the  feedback  feature  is  turned  on,  the 
tasking  evaluations  window  is  displayed  immediately  after  the  user  transmits  mission  taskings  to  a  WOC. 
Summary  measures  can  be  displayed  at  any  time.  These  summary  measures  consist  of  histograms  of  air 
request  disposition  and  mission  results,  graphs  of  asset  allocations  to  Army  divisions  (fire  priorities),  and 
other  numerical  measures  expressed  using  text,  level  meters,  and  strip  charts. 

Support  Tools 

System  support  tools  include  software  for  building  exercises.  A  building  interface  and  building 
functions  allow  system  developers  and  researchers  to  adjust  SIMSIM  object  attributes  for  the  SIMPLE, 
INTERMEDIATE,  and  COMPLEX  game  levels. 
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Using  SuperKEATS  for  Student  Practice 


A  student  user  acquires  and  improves  decision-making  skills  through  repeated  game  playing  in  the 
reactive,  simulated  microworld.  Student  actions,  system  reactions,  and  simulation  probabilities  make  each 
gaming  session  unique. 

The  student  first  selects  a  game.  The  SuperKEATS  game  then  unfolds  in  two  phases:  an  initial 
orientation  phase  and  a  subsequent  operations  phase. 

Beginning  the  orientation  phase,  SuperKEATS  prompts  the  student  to  assess  the  tactical  situation  and 
to  develop  a  macro-level  plan  of  action.  After  the  student  implements  an  initial  plan,  he  starts  the  game 
clock  and  enters  the  operations  phase.  During  the  operations  phase,  the  student  makes  micro-level 
decisions  and  replans  at  the  macro-level  in  response  to  incoming  tactical  messages.  We  discuss  these 
actions  below  in  greater  detail,  referencing  the  actions  in  Figure  7. 

Game  Selection 

The  student  selects  from  among  the  SuperKEATS  games  built  by  the  SuperKEATS  researcher.  Three 
basic  SuperKEATS  games  were  developed  to  support  the  initial  SuperKEATS  evaluation.  The  simple 
game  offers  a  single  type  of  aircraft  asset,  with  no  loss  of  assets  during  the  game.  The  intermediate  game 
provides  three  asset  types  and  some  loss  of  assets  due  to  air  attacks  on  bases  and  air  defense  attacks  on 
missions.  The  complex  game  features  six  asset  types,  an  added  air  request  type,  and  asset  losses  resulting 
from  increased  probabilities  of  air  attacks  on  bases,  small  unit  ground  attacks  on  bases,  and  air  defense 
attacks  on  missions.  A  researcher  may  add  many  more  games  to  this  set  to  experiment  with  how  games 
can  be  built  and  sequenced  to  meet  specific  training  objectives. 

Orientation  Phase 

Situation  Study.  During  the  orientation  phase  of  a  game,  the  student  first  familiarizes  himself  with 
the  overall  tactical  situation.  This  initial  step  simulates  the  beginning  of  the  day  shift  in  the  real  world. 
This  familiarization  process  is  enabled  by  making  available  the  same  information  resources  that  are 
present  in  the  real  world: 

1.  Information  summaries  for  corps,  divisions,  bases,  and  squadrons. 

2.  Status  boards  for  base,  asset  allocation,  and  mission. 

3.  ASOC  briefings  and  orders. 

4.  Factual  information  on  aircraft  types,  ordnance  types,  and  ordnance  loads. 

5.  A  map  of  the  tactical  situation. 
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ORIENTATION 


SuperKEATS  Gaming  Session  Flow 


Situation  Assessment.  Much  as  the  FDO  does  in  the  ASOC,  the  student  can  formulate  a  “big 
picture”  assessment  which  anticipates  the  air  support  needs  of  the  ground  forces  for  the  day  and 
accurately  accounts  for  aircraft  assets  by  performing  a  thorough  situation  study.  This  initial  situation 
assessment  in  turn  leads  to  an  initial  plan  of  action. 

Plan  of  Action.  The  plan  assigns  air  assets  to  primary  and  alternate  Army  divisions  in  either  an  alert 
or  smooth  flow  mode.  Because  SuperKEATS  realistically  simulates  the  delays  necessary  to  prepare  assets 
for  missions.it  is  critical  that  the  student  first  formulate  and  implement  such  a  plan.  Alerts  prepare  air 
assets  for  immediate  tasking  by  putting  them  on  various  levels  of  readiness.  Smooth  flows  preplan  mission 
times  in  “airline  schedule”  style  for  predetermined  time  blocks  and  mission  intervals. 

The  student  enters  the  situation  assessment  and  plan  of  action  on  an  Initial  Assessment  and  Plan  form 
(Figure  8). 
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Figure  8.  A  Sample  Assessment  and  Partial  Game  Plan. 


Air  Asset  Preparation.  The  student  then  begins  to  implement  the  plan  by  telling  WOCs  to  put 
aircraft  on  alert  or  to  schedule  a  flow  of  missions,  specifying  additional  details  such  as  timing  and  numbers 
of  aircraft.  During  game  playing,  the  student  receives  suggestions  and  feedback  based  on  the 
implemented  plan.  If  the  student  neglects  to  implement  a  plan  or  implements  a  deficient  plan  before 

running  the  game,  the  simulation  will  provide  appropriate  feedback.  For  example,  if  the  student  does  not 

« 

put  aircraft  on  alert  before  starting  the  game,  the  WOC  will  refuse  mission  taskings  for  requests  received 
early  in  the  game.  The  student  can  take  as  much  time  as  he  wants  to  complete  the  orientation  phase. 

Set  Playing  Mode  and  Start  Game  Clock 

The  student  begins  a  game  and  enters  the  operations  phase  by  setting  the  playing  mode  to  one  of  four 
modes  and  pushing  the  NEXT  button  to  start  the  game  clock.  The  playing  modes  are:  a  pushbutton  mode 
which  advances  the  clock  whenever  the  student  pushes  the  NEXT  button  and  the  real-time  modes  which 
automatically  advance  the  clock  at  1,  2,  or  10  times  real  time. 

Operations  Phase 

During  the  operations  phase  of  the  game,  the  student  reacts  to  events  as  he  or  the  SuperKEATS 
simulation  triggers  them  over  time.  Events  show  up  as  ASOC-received  messages  displayed  in  a  message 
log.  On  the  macro  level,  the  student  may  reassess  the  overall  tactical  situation  for  the  ASOC  and  may 
decide  to  change  the  plan  of  action  to  one  which  is  consistent  with  the  new  assessment.  On  the  micro 
level,  the  student  decides  what  to  do  about  a  specific  event.  Decisions  on  this  level  are  guided  and 
constrained  by  decisions  made  on  the  planning  or  macro  level. 

For  example,  the  student  receives  an  air  request  event  message  at  7:16  a.m.  as  a  brief  message  from 
the  Blue  Division  3  ALO  to  neutralize  15  tanks  as  soon  as  possible  (ASAP),  and  not  later  than  (NLT) 
8:15  a.m.  He  may  also  receive  SuperKEATS-generated  mission  tasking  suggestions  and  options  at  the 
same  time  to  aid  his  decision-making.  He  then  reacts  to  events  as  he  would  in  the  real  world  by  accessing 
various  information  sources  to  confirm  the  event,  to  gather  additional  information  about  it,  and  to 
determine  solution  options.  After  this  period  of  “cogitation,"  he  enters  the  event  decision  input  by  telling 
a  WOC  to  task  four  alert  aircraft  from  squadron  TFSll  to  meet  the  air  request.  If  artificial  feedback  is 
turned  on,  he  receives  an  evaluation  of  the  event  decision  input  based  on  a  comparison  with  the 
SuperKEATS-generated  suggestions  and  options.  The  event  decision  is  then  input  to  the  simulation. 
SuperKEATS’  simulation  evaluates  the  student’s  input  by  checking  for  violation  of  physical  constraints. 
SuperKEATS  checks  whether  these  airplanes  are  indeed  available  at  that  squadron  and  whether  they  can 
make  the  desired  NLT.  The  results  of  constraint  checking  are  returned  to  the  student  as  feedback 
messages.  If  the  tasked  mission  does  not  violate  any  physical  constraints,  the  user  receives  a  message 
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which  indicates  that  the  WOC  accepts  the  tasking.  Otherwise,  the  WOC  sends  a  refusal  message  to  the 
student.  The  student  can  elect  to  display  a  game  summary  displaying  his  overall  performance  at  any  time 
during  the  operations  phase.  The  student  may  also  reassess  the  t~~t<eal  situation  and  decide  to  implement 
a  new  plan;  for  example,  the  request  may  confirm  the  student’s  assessment  of  an  offensive  push  against 
Blue  Division  3  and  cause  him  to  schedule  a  smooth  flow  of  missions  to  Blue  Division  3,  thereby  revising 
his  plan  to  use  alert  aircraft  to  support  Blue  Division  3. 

The  individual  steps  constituting  the  operations  phase  are  described  below. 

Event  Presentation  Message.  As  simulation  dme  advances,  the  student  learns  of  the  occurrences  of 
events  through  messages  displayed  in  the  message  log  (Figure  9) .  If  the  event  is  an  air  request,  the  target 
position  associated  with  the  request  is  plotted  on  the  map.  The  student  can  also  examine  data  on  the  air 
request  itself. 
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Figure  9.  Typical  Student  Display  for  Playing  a  Game. 
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Suggestions  and  Options.  Suggestions  and  options  may  be  turned  on  or  off.  Researchers  may 
choose  on  for  a  novice,  off  for  an  advanced  student,  or  leave  it  up  to  the  student.  A  novice  student  may 
receive  both  tasking  suggestions  and  options  for  maximal  SuperKEATS  guidance  (Figure  10).  A 
proficient  student  may  use  the  same  feature  as  a  decision  aid  which  keeps  track  of  the  feasible  set  of 
options.  An  expert  student  may  choose  not  to  use  the  feature.  In  any  case,  whenever  the  suggestions  and 
options  are  turned  on  and  an  air  request  is  received,  the  Tasking  Suggestions/Options  display  gives  the 
student  guidance  and/or  specific  tasking  options  for  the  air  request.  The  student  may  also  access 
suggestions  and  options  via  the  AIR  REQUESTS  display  for  a  particular  air  request. 

Situation  Study.  For  further  information,  the  student  can  request  and/or  monitor  information  from 
the  same  sources  that  were  available  in  the  orientation  phase  (i.e.,  ASK,  STATUS  BOARDS,  and  INFO). 
All  status  boards  update  automatically  and  continuously  as  bases  close  and  reopen,  aircraft  take  off  and 
return,  aircraft  change  status,  and  aircraft  are  lost.  Briefings,  orders,  and  factual  information  never 
change  during  a  game. 


Figure  10.  Sample  Tasking  Suggestions  and  Options  Display. 
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Event  Decision  Input.  When  the  student  decides  on  assets  to  task  for  a  request,  the  student  tells  a 
WOC  to  perform  an  action  via  a  tasking  message  (Figure  11).  The  message  references  the  air  request 
number  and  specifies  the  number  of  aircraft,  the  aircraft  type,  the  air  unit,  and  the  aircraft  pool.  The 
information  content  of  the  message  is  fundamentally  the  same  as  that  transmitted  by  an  FDO  in  the 
operational  environment.  When  the  message  is  complete,  the  student  transmits  the  message  to  the 
WOC. 


Figure  11.  Sample  Mission  Tasking  Display. 

Artificial  Feedback.  Artificial  feedback  may  be  turned  on  or  off.  The  feedback  option  gives 
immediate  feedback  on  an  air  request  tasking  decision  (Figure  12).  When  the  student  tells  the  WOC  to 
task  a  mission,  the  feedback  is  first  displayed  instead  of  an  immediate  WOC  response.  The  student  then 
has  the  option  to  either  change  the  decision  or  forward  it  on  for  a  WOC  response. 
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Figure  12.  Sample  Tasking  Evaluations  Display. 
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Event  Decision  Input  to  Simulation.  The  immediate  natural  feedback  comes  in  the  form  of 
acceptance/refusal  responses  from  the  WOC  which  are  immediately  displayed  as  messages  in  the  message  log. 
The  WOC  performs  feasible  actions  and  refuses  impossible  ones.  All  affected  Status  Board  displays  are 
then  automatically  updated. 

Additional  feedback  comes  from  the  simulation  after  a  typically  expected  time  delay.  For  example, 
air  support  mission  results  are  displayed  in  the  message  log  as  an  in-flight  report  transmitted  as  the  mission 
aircraft  return  to  base.  The  student  can  easily  associate  the  feedback  with  the  tasking  decision.  The  other 
longer-term  consequences  of  the  tasking,  such  as  changes  in  Army  force  ratios,  division  movement,  and 
air  request  generation,  are  more  difficult  to  directly  correlate  with  a  particular  tasking  decision.  However, 
the  student  may  receive  additional  requests  for  air  support  and  notice  trends  in  either  target  sizes  (the 
number  of  tanks  targeted)  or  target  positions. 

Game  Summary.  A  game  summary  (Figure  13)  displays  histograms  of  air  request  disposition  and 
tasking  results  and  gaming  effectiveness  measures  (GEMS).  Additional  displays  include: 

!•  Graphs  comparing  Corps  division  priorities  with  ASOC  taskings. 

2.  An  explanation  of  GEMS. 

3.  GEMS  Meters. 

4.  GEMS  Plots. 
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Figure  13.  Sample  Game  Summary  Display. 
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GEMS  are  nonjudgmental,  quantitative  scores  for  student  performance  during  a  game.  SuperKEATS 
calculates  GEMS  ratios  using  completed  missions  data  (Table  4). 

Table  4.  Explanation  of  Gaming  Effectiveness  Measures  (GEMS) 

1.  AE  =  Air  Effectiveness  =  Tanks  Neutralized/Tanks  Targeted.  The  greater  the  value  of  AE 
the  better.  AE  sometimes  exceeds  1  when  reserves  are  neutralized  in  addition  to  the  tanks 
originally  targeted  on  the  air  request. 

2.  SEf  =  Sorties  Effectiveness  =  Tanks  Neutralized/Sorties  Total.  The  greater  the  value  of  SEf 
the  better.  This  value  reflects  the  aircraft  type  mix,  the  presence  of  Red  air  defenses,  and 
air/ground  coordination. 

3.  SEc  =  Sorties  Economy  =  Sorties  Total/Tanks  Targeted.  A  low  SEc  shows  economical  use 
of  sorties. 

4.  ER  =  Exchange  Ratio  =  Sorties  Lost/Tanks  Neutralized.  A  low  ER  is  desirable.  Higher 
ERs  occur  with  poor  mission  configuration,  good  Red  air  defenses,  and  lack  of  air/ground 
coordination. 

5.  Ut  =  Utilization  =  Sorties  Total/Sorties  Projected.  Ut  should  ideally  approach  100%  at  the 
end  of  the  game  to  show  full  utilization  of  all  available  resources. 

GEMS  meters  and  plots  are  separate,  compact  displays  which  support  continuous  performance 
monitoring.  GEMS  meters  consist  of  a  stack  of  five  small  meter  boxes  showing  the  current  GEMS  levels. 
GEMS  plots  consist  of  a  stack  of  five  small  strip  charts  which  plot  past  and  current  GEMS  values.  GEMS 
meters  and  plots  are  updated  automatically  whenever  mission  results  are  reported. 

The  student  or  researcher  may  request  a  full  game  summary  or  display  GEMS  meters  and  GEMS  plots 
at  any  point  during  game  playing. 

Using  SuperKEATS  for  Researcher  Experimentation 

A  researcher  builds  and  modifies  SuperKEATS  games  and  controls  SuperKEATS  instructional 
parameters  to  evaluate  their  effect  upon  training.  Instructional  macro-strategies  are  easily  tuned  by 
adjusting  game  complexity  and  introducing  new  games.  During  gaming  sessions,  a  researcher  formulates 
micro-strategies  for  instruction  by  turning  instructional  displays  on  and  off,  and  adjusting  the  game  pace. 
The  researcher  may  also  permit  the  student  user  to  manage  the  displays  and  the  pace  of  the  game 
exercise. 
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Building  Games 


The  ISD  researcher  can  build  tailored  games  to  achieve  desired  training  objectives  using  the 
SuperKEATS  exercise  building  tools.  Table  5  lists  the  differences  in  complexity  parameters  among  the 
currently  developed  games  which  represent  the  three  gaming  levels. 


Table  5.  Game  Complexity  Parameters 


Gaming  Levels 

Parameter 

Effect 

Simple 

Intermediate 

Air  status 

Base  air  attacks 

Supremacy 

Superiority 

Parity 

Small  Unit  Attack 
Capability 

Base  ground  attacks 

False3 

False3 

Trueb 

Joint  Air  Attack 

Team  (JAAT) 
Capability 

Type  of  requests 

Air  support 
effectiveness 

False3 

False3 

Trueb 

Red  Air  Defense 

Aircraf.  losses 

Nonec 

Goocf1 

Goodd 

Number  of  WOCs 

Aircraft  types 

1 

3 

6 

Base  Defense 

Forces 

Base  attack 
effectiveness 

Truee 

True® 

f 

Varies* 

Number  of  Turns 

Asset  use/ 
availability 

Same8 

Varies*1 

Varied1 

a False  implies  that  the  capability  does  not  exist. 
bTrue  implies  that  the  capability  exists. 

'None  implies  that  there  are  no  Red  Air  Defenses. 
dGood  implies  that  the  Red  Air  Defense  exists  and  its 
capability  is  good. 

eTrue  implies  that  the  bases  all  have  defense  forces. 


f  Varies  means  that  bases  may  or  may  not  have  defense 
forces  —  the  parameter  is  set  for  each  base  in  a 
particular  game. 

8  Same  means  that  all  aircraft  will  have  same  number  of 
turns. 

h  Varies  means  that  squadrons  will  have  assigned  turns 
depending  on  the  exercise. 


The  researcher  can  change  the  complexity  by  building  new  games  with  different  values.  The 
researcher  can  regulate  the  length  of  a  game,  its  start  time,  and  the  frequency  of  simulation  updates. 
Games  can  be  a  mini-game  covering  a  single  time  block,  or  a  full  game  covering  an  entire  day. 

In  addition  to  the  parameters  above,  the  division  priorities  of  fire,  the  corps/division/company 
objectives,  the  numbers  of  tanks  in  companies,  and  the  numbers  of  tanks  in  reserve  can  all  be  modified  to 
produce  dramatically  different  games.  Existing  SuperKEATS  games  all  put  the  Blue  Corps  on  the 
defensive;  the  researcher  can  build  offensive  scenarios. 

The  researcher  can  change  the  complexity  by  building  new  games  with  different  values.  The 
researcher  can  regulate  the  length  of  a  game,  its  start  time,  and  the  frequency  of  simulation  updates. 
Games  can  be  a  mini-game  covering  a  single  time  block,  or  a  full  game  covering  an  entire  day. 
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On  the  asset  side,  there  may  be  up  to  10  WOCs  with  3  squadrons  each.  Each  squadron  has  an 
associated  aircraft  type,  number  of  aircraft,  and  base  location.  Furthermore,  up  to  10  bases  with  differing 
aircraft  maintenance  capabilities  may  be  created. 

Instructional  Feature  Control 

Throughout  a  gaming  session,  suggestions  and  options,  feedback,  and  GEMS  may  be  turned  on  and 
off.  The  researcher  can  explore  individual  student  preferences  for  the  features,  the  student’s  utilization  of 
the  features,  and  the  instructional  effectiveness  of  the  features.  Variables  in  this  exploration  can  include 
the  student’s  level  of  skill  acquisition  and  learning  style. 

III.  RESULTS 

SuperKEATS  has  been  used  once  by  ASOC  personnel  during  this  project  phase.  The  methods  used 
during  this  trial  and  the  results  obtained  are  reported  here. 

Evaluation  Method 

Two  712th  ASOC  personnel  used  the  system  over  a  2-day  period  at  the  contractor’s  facility.  One 
subject  was  an  8-year  veteran  of  the  712th  ASOC,  with  experience  in  all  aspects  of  the  FDO  job;  the 
other  had  only  1  year  of  limited  experience,  having  never  worked  the  day  shift  during  an  exercise.  This 
difference  in  experience  presented  the  opportunity  to  compare  novice  and  expert  gaming  behavior  and 
effectiveness. 

Each  subject  used  the  system  for  1  day.  Neither  subject  was  present  while  the  other  used  the  system. 
First,  contractor  personnel  instructed  the  subject  on  system  use.  Second,  the  subject  examined  a  simple 
game  to  achieve  system  familiarity.  Third,  the  subject  played  a  complex  game  to  evaluate  the  system. 
Finally,  at  the  end  of  the  day,  the  subject  completed  an  evaluation  form. 

Gaming  Results 

The  expert  processed  more  than  twice  as  many  events  as  the  novice  while  playing  the  complex  game 
(see  Table  6).  The  expert  focused  on  the  FDO  task  and  sorted  in  the  real-time  mode  then  moved  to  10 
times  the  real-time  mode.  The  novice  explored  interface  features  while  playing  in  real  time. 
SuperKEATS  actually  ran  slower  than  real  time  as  the  game  progressed. 

The  elapsed  game  minutes  do  not  include  the  time  both  subjects  devoted  to  the  initial  assessment, 
planning,  and  setup  phase.  These  times  are  included  in  real-time  hours.  Appendix  B  contains  the 
protocol  of  the  expert  playing  the  SuperKEATS  complex  game. 
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Table  6.  Complex  Game  Results 


Novice 

Expert 

Elapsed  game  minutes 

124 

249 

Real-time  hours 

4 

5 

Message  log  pages 

8 

20 

Macro-Level  Performance 

Both  subjects  began  the  complex  game  with  the  same  tactical  situation.  The  expert  produced  a  9-line 
initial  situation  assessment  (ISA)  and  a  plan  of  action  (POA) .  The  novice  did  not  enter  an  assessment  but 
did  complete  a  POA. 

An  analysis  of  the  expert’s  ISA  found  common  content  elements  with  previously  analyzed  KEATS 
exercises  (Brecke  et  al.,  1989)  in  three  areas:  statements  defining  the  probable  demand  for  air  support, 
statements  evaluating  the  anticipated  supply  of  air  resources,  and  identification  of  replanning  triggers  or 
factors  that  should  be  watched  because  they  might  trigger  a  revision  of  the  initial  plans  (see  Table  7). 

Plans  consisted  of  a  mapping  of  air  resources  by  either  wings  or  squadrons  to  front-line  divisions.  The 
two  POAs  are  shown  in  graphic  form  in  Figure  14.  They  represent  clearly  very  different  ways  of  dealing 
with  the  same  situation. 


Table  7.  Content  Elements  in  Situation  Assessment 


Element 

Expert  using 
SuperKEATS 

Expert  using 
KEATS 

Demand  assessment 

Corps  objectives 

X 

X 

Division  objectives 

X 

X 

Attack  times 

X 

X 

Corps  priorities 

X 

X 

Supply  assessment 

Type  of  a/c 

X 

X 

Ordnance 

X 

Preplanned  allocation3 

X 

Time  to  target  area 

X 

Replanning  triggers 

X 

X 

No  preplanned  allocations  in  SuperKEATS. 


EXPERT 

NOVICE 

WING  DIVISION 

WING  DIVISION 

-l£*!£*AFT 


L*!*crafj 


CD 


GY 


UWCSiC— 9>\  (  ^ 
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CD 
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4  SMOOTH  FLOWS 


NOTES: _ 
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Figure  14.  Plan  of  Action  Differences  in  Identical  Situation. 


36 


Both  subjects  implemented  their  initial  plan  before  starting  the  game.  After 

same  minutes,  the 

novice  cancelled  the  Wing  E  smooth  flow  to  Division  1  and  then  unsuccessfully  attemotPH  . 

y  a  to  put  those 

aircraft  on  alert.  He  put  eight  more  Wing  A  aircraft  on  alert  status  30  game  minutes  l*er>  expert  dlQ 
not  implement  any  alert  changes  nor  did  he  initiate  any  smooth  flows  during  his  gan^g  session 

The  expert  tasked  92%  (166  out  of  180)  of  sorties  to  primary  and  alternate  divisic\s>  compared  to 
89%  (80  out  of  90)  for  the  novice  (Table  8).  Of  the  sorties  directly  tasked  by  the  novice,  out  of 

30)  adhered  to  his  plan. 


Table  8.  Sorties  Tasked  According  to  Plan 


Novice  after 

119  minutes 

Expert  after 

249  minutes 

Smooth  flow  sorties 

60 

0 

Directly  tasked  sorties 

30 

180 

Directly  tasked  to  primary 

12 

140 

Directly  tasked  to  alternate 

8 

26 

Sorties  Aborted 

4 

0 

Sorties  Diverted 

4 

0 

Micro-Level  Performance 

Gaming  sessions  for  both  subjects  included  the  same  typical  events:  The  same  two  bases  were  attacked 
at  the  start  of  the  game,  resulting  in  identical  losses  and  base  closure  times.  Subsequent  aircraft  losses 
occurred  only  when  flying  missions. 

Air  request  statistics  (Table  9)  were  similar  only  in  terms  of  system  generation  rates  and  targets.  The 
novice’s  smooth  flow  plan  substantially  reduced  the  number  of  requests  received  by  the  ASOC. 


Table  9.  Air  Request  Statistics 


Novice 

Expert 

Request  Generated/Game  Hour 

8.5 

7.75 

Number  of  Target  Tanks/Request 

20 

26 

Number  Received  by  ASOC/Game  Hour 

1.5 

7.75 

„  vision  movement  demonstrated  the  expert’s  tasking  prowess  (Table  10).  The 

Gaming  measures 

mad  twice  »s  many  tanks  per  request  (AE)  and  per  sortie  (SEf)  as  did  the  novice.  This 
expert  neutral 

suing  effectiveness  w'S  most  evident  in  the  Division  1  sector,  where  the  expert  held  the  attack  to  a 
1-kilometer  advance  -ompared  to  the  novice’s  10  kilometers.  An  examination  of  game  protocols  showed 
that  the  novice’s  faillre  to  use  m°re  effective  aircraft  to  supplement  the  smooth  flow  sorties  in  the  Division 
1  area  was  a  ma]V  contributing  factor. 

parties  Economy  (SEc)  and  Exchange  Ratio  (ER)  measures  were  comparable  for  both  subjects, 
indicating  theiConorn'cal  use  of  sorties  with  minimal  inflight  sortie  losses  to  achieve  tank  neutralization. 
Sortie  utilization  (Ut)  was  conservative  for  both  subjects  if  compared  to  an  expected  linear  utilization  rate 
of  9%  per  each  of  11  game  hours.  However,  the  expert,  after  4  hours  of  play,  was  only  5%  below  the 
expected  linear  value.  This  can  be  explained  in  part  by  flight  time  delays  at  the  start  of  the  game  in  that 
Ut  accounts  only  for  sorties  that  have  actually  reached  the  target  and  not  for  tasked  sorties  in  preparation 
or  enroute. 

Table  10.  Gaming  Measures3 and  Division  Movement  Comparisons 


Number  of  Requests 

Tanks  Neutralized 
Tanks  Targeted 
Sorties  Over  Target 
Sorties  Lost 

Air  Effectiveness  (AE) 

Sorties  Effectiveness  (SEf) 
Sorties  Economy  (SEc) 
Exchange  Ratio  (ER) 

Utilization  (Ut) 

Enemy  Advances  in  Kilometers 
Division  1 
Division  2 
Division  3 


Novice  after 

119  minutes 

Expert  after 
141  minutes 

Expert  after 

249  minutes 

10 

14 

25 

44 

160 

279 

200 

299 

634 

44 

88 

144 

2 

5 

11 

0.22 

0.54 

0.44 

1.00 

1.82 

1.94 

0.22 

0.29 

0.23 

0.05 

0.03 

0.04 

10% 

19% 

31% 

10 

1 

1 

0 

0 

0 

0 

0 

0 

Written  Evaluation  Results 


Overall,  SuperKEATS  was  judged  useful  to  the  ASOC,  with  an  overall  high  rating  of  4  on  a  scale  of  1 
to  5.  The  expert  particularly  liked  the  ACTION  REACTION-RESULT  chain  introduced  by 
SuperKEATS,  underscoring  the  achievement  of  the  major  SuperKEATS  objective  to  furnish  a  reactive 
environment  which  was  lacking  in  KEATS.  The  expert’s  evaluation  is  provided  in  Appendix  C.  Both 
expert  and  novice  disliked  display  functionality,  recommending  more  compact  and  complete  information 
displays.  Both  subjects  encouraged  further  development  and  suggested  the  use  of  SuperKEATS  in 
operational  exercises. 

Fidelity  of  Simulation 

Responses  were  generally  favorable  (4  on  a  scale  of  1  to  5)  for  inclusion  of  expected  objects,  events, 
availability  of  information,  and  ability  to  communicate  with  other  agents.  Extensions  suggested  include: 

1.  Weather  events. 

2.  Maintenance  downtime  events. 

3.  Longer  base  closures. 

4.  Requests  with  non-worthwhile  targets  for  FDO  to  reject. 

5.  Quick  access  to  available  aircraft  for  tasking. 

6.  Communications  with  G-3  Air  concerning  priorities. 

The  simulation  response  to  user  decisions,  system  response  times,  and  information  display 
management  received  lower  ratings.  Of  special  concern  was  the  replacement  time  for  ground  force  losses. 
During  the  gaming  session,  the  simulation  replaced  losses  within  hours  instead  of  the  expected  day  or 
days.  Also,  the  initial  conditions  for  the  complex  game  were  set  to  initially  generate  requests  from  all 
divisions  rather  than  the  expected  concentration  of  requests  from  one  area  then  another.  The  expert 
thought  that  losses  were  instantly  replaced  and  that  the  replacement  “did  not  allow  the  FDO  to  change 
focus  to  another  area.”  The  expert  recommended  replacement  of  losses  in  unit  increments  over  longer 
time  frames. 

A  large  number  of  targets,  such  as  72  tanks,  on  an  air  request  was  judged  unrealistic  and  sortie 
effectiveness  was  thought  too  high  with  up  to  eight  targets  neutralized  by  one  sortie.  (A  final  release 
modification  to  SuperKEATS  has  since  reduced  sortie  effectiveness.) 

Both  subjects  indicated  trouble  with  arranging  information  displays  to  facilitate  quick  and  easy  access. 
Both  users  seemed  to  prefer  a  functional  rather  than  a  physical  replication  of  ASOC  sources  of 
information  to  support  gaming  operations.  A  dual  screen  or  larger  display  was  recommended. 


39 


SuperKEATS  for  Training 


A  Novice’s  Perspective.  The  novice  awarded  the  SuperKEATS  feedback  feature  his  highest  rating 
and  gave  training  coverage,  validity  and  efficiency  of  training,  and  the  suggestions/options  feature  lower 
ratings.  He  commented  that  SuperKEATS  did  not  reinforce  procedural  actions  that  a  FDO  is  expected  to 
perform  and  did  not  cover  European  ASOC  operations.  He  suggested  an  expansion  of  the  suggestions 
and  options  feature  to  include  additional  information  such  as  base,  WOC,  alternate  selection,  missions  to 
divert,  retasking,  and  smooth  flow  options. 

An  Expert’s  Perspective.  SuperKEATS  received  overall  high  ratings  for  training  a  novice  in  all 
aspects  except  for  coverage.  The  expert  recommended  improvements  to  simulation  fidelity  that  were 
closely  coupled  with  training.  Elements  he  considered  important  for  training  a  novice  included: 

1.  Events/requests  that  would  be  disapproved. 

2.  Actions  to  have  or  not  have  positive  effects  in  particular  sectors  (allowing  shifting  of 
emphasis/concern) . 

3.  Closing  bases  for  at  least  a  period  of  time  that  would  affect  request  servicing,  an  hour  instead  of 
a  half  hour. 

He  added  that  more  complex  events  would  be  required  for  training  an  experienced  FDO. 

The  expert  liked  the  suggestions/options  and  tasking  feedback  features  for  the  novice  learner.  He 
wrote  that  suggestions/options  could  be  a  useful  tool  for  real-world  tasking.  He  also  wrote  that  feedback 
could  be  used  for  experienced  users  after  reacting  to  complex  events,  as  a  means  to  review  all  areas  to  be 
considered. 

IV.  DISCUSSION  AND  CONCLUSIONS 

In  this  section,  SuperKEATS  is  discussed  from  two  perspectives:  its  relationship  to  earlier  technologies 
for  training  decision-making  skills,  and  lessons  learned  in  building  and  using  it.  SuperKEATS  has 
received  a  minimal  amount  of  use  by  ASOC  personnel  during  this  project  phase.  Hence  a  discussion  of 
the  effectiveness  of  SuperKEATS  for  training  decision-making  skills  and  for  instructional  strategy 
experimentation  awaits  further  work. 

Comparison  with  Earlier  Microcomputer  Systems 

In  Section  I,  earlier  work  on  small-scale,  inexpensive  decision-training  systems  was  referenced.  The 
question  now  is  how  SuperKEATS  compares  with  these  earlier  systems  in  terms  of  scope,  objectives,  and 
domain.  In  terms  of  scope  and  objectives,  only  SuperKEATS  has  had  training  as  its  main  objective.  It 
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provides  training  on  the  macro  level  by  allowing  modification  of  exercise  complexity,  and  on  the  micro 
level  by  using  prompting  and  artificial  feedback.  The  work  of  Madni,  Ahlers,  and  Chu  (1987)  focused 
mainly  on  the  modeling  of  an  intelligent  adversary  and  allowed  modification  of  the  model  as  a  means  to 
train  various  objectives.  Obermayer,  Johnston,  Slemon,  and  Hicklin  (1984)  showed  that  modeling  with 
an  expert  system  could  reduce  the  labor  intensity  of  training,  but  the  “Game  Environment  Simulator”  they 
built  was  a  simulation  and  not  intended  as  a  training  system.  Wilson  (1982)  created  a  wargame  that  might 
have  training  utility  as  a  secondary  feature.  Stoddard,  Kern,  and  Emerson  (1986)  had  a  long-term  goal 
of  producing  an  intelligent  tutoring  system  for  tactical  decision-making  but,  at  this  stage,  report  only  the 
design  for  a  knowledge  extraction  tool,  serving  a  purpose  much  like  KEATS,  without  explicit  training 
features. 

In  terms  of  the  domain,  SuperKEATS  attempted  to  model  a  more  complex  domain  than  have  any  of 
the  earlier  systems,  although  it  does  impose  limitations  on  weather  modeling  and  terrain.  SuperKEATS 
needed  to  model  Army  unit  actions  from  the  corps  level  down  to  the  front-line  company,  Air  Force  units 
from  the  wing  operation  center  to  individual  aircraft,  and  the  interaction  of  air  units  attacking  ground 
targets.  Madni  et  al.  (1987)  simulated  a  one-on-one  interaction  of  two  surface  ships.  Obermayer  et  al. 
(1984)  had  a  complex  environment  with  ships,  a  helicopter,  a  submarine,  sonobuoys,  torpedos,  etc.,  but 
the  game  was  based  only  on  anti-submarine  warfare  (ASW)  concepts  and  did  not  attempt  fidelity  to  a  real 
ASW  environment.  Wilson  (1982)  modeled  only  the  air  battle  and  Stoddard  et  al.  (1986)  were  interested 
in  the  Army  armor  officer  directing  platoons  of  tanks  in  a  land  battle. 

SuperKEATS,  then,  advances  the  technology  base  for  decision-making  training  by  tackling  a  complex 
domain  intent  on  producing  a  training  system  with  real-world  operational  use,  where  completeness  and 
fidelity  are  key  issues.  That  it  has  succeeded  is  evidenced  by  the  positive  responses  of  the  ASOC  members 
who  have  used  it. 


Comparison  with  an  Intelligent  Tutoring  System 

SuperKEATS  is  not,  and  was  not  intended  to  be,  an  intelligent  tutoring  system  (ITS).  Because 
SuperKEATS  does  provide  an  automated  environment  for  practicing  (and  learning)  decision-making 
skills,  a  comparison  with  an  ITS  puts  the  capabilities  of  SuperKEATS  in  perspective.  In  this  discussion, 
the  ITS  for  a  gaming  environment  (Burton  &  Brown,  1982;  Goldstein,  1982)  will  be  used  for  comparison. 

Typically,  an  ITS  will  have  a  model  of  an  expert  that  generates  responses  and  solutions  that  are 
compared  with  the  student’s  responses.  Although  a  stand-alone  expert  model  was  not  built  for 
SuperKEATS,  rules-of-thumb  and  generation  of  tasking  options  are  implemented  that  use  rules  obtained 
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from  ASOC  experts.  A  key  difference  between  the  complex  ASOC  environment  and  most  previous  ITS 
environments  is  the  existence  of  multiple  acceptable  solutions  for  air  asset  tasking.  It  was  important  for 
prompting  that  all  reasonable  options  be  presented  and,  for  feedback,  that  choosing  a  reasonable  option 
be  accepted.  Hence,  an  expert  ASOC  FDO  model,  as  such,  that  always  arrived  at  a  specific  tasking 
solution,  was  not  necessary  or  useful  for  training  in  this  environment. 

SuperKEATS  does  not  provide  a  tutor/instructor  model  as  would  be  present  in  an  ITS  to  direct,  at  the 
macro  level,  the  topics  of  instruction,  and  at  the  micro  level,  to  set  the  level  and  amount  of  prompting  and 
feedback.  SuperKEATS  also  does  not  provide  a  student  model  (also  expected  in  an  ITS)  to  keep  track  of 
the  student’s  current  level  of  expertise  and  topics  learned.  Viewed  from  its  perspective  as  the  practice 
environment  in  a  larger  decision  training  system,  these  tutoring  and  student  model  capabilities  would,  in 
one  form  or  another,  have  to  be  included  in  this  larger  system,  of  which  SuperKEATS  is  a  part. 

Stability  of  the  Environment 

A  major  concern  with  any  automated  system  is  its  brittleness  as  the  world  for  which  it  was  built 
changes.  To  try  to  build  an  automated  system  that  can  .cope  with  any  changes  in  its  area  of  operation 
would  be  very  expensive,  if  even  possible.  If  the  environment  is  changing  so  rapidly  that  between  the  time 
the  system  is  designed  and  completed  it  has  become  obsolete  or  at  least  needs  major  upgrades,  then 
building  the  system  in  the  first  place  must  be  questioned.  Among  the  major  concerns  with  Intelligent 
Tutoring  Systems  (ITSs)  is  how  difficult  they  are  to  build  and  then  how  easily  they  fail  if  the  environment 
changes.  Because  SuperKEATS  was  built  as  a  simulation  practice  environment  and  is  an  alternative  to  a 
traditional  ITS,  its  brittleness  is  of  interest.  With  SuperKEATS,  brittleness  depends  on  the  type  of 
change,  and  various  possibilities  are  discussed  below. 

Changes  to  equipment  such  as  aircraft  or  ordnance  that  affect  only  model  parameters  can  be  easily 
accommodated  in  SuperKEATS.  The  changed  parameters  are  readily  modified  in  the  “fact”  knowledge 
base.  If  equipment  changes  affect  the  suggestion/feedback  rules  about  when  to  use  the  equipment,  then 
changes  to  the  rules  will  need  to  be  made.  These  rule  changes  must  be  based  on  expert  knowledge  and 
will  likely  be  somewhat  involved. 

Changes  in  the  ASOC  tactics  that  have  been  captured  in  the  suggestion/feedback  rules  will  likewise 
require  changes  in  the  rules.  If  the  change  is  fairly  basic,  then  the  rules  may  need  be  acquired  anew, 
possibly  by  using  KEATS.  If  the  change  is  minor,  it  may  affect  only  the  conditions  and  actions  of  a 
limited  group  of  rules. 
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Changes  to  the  functions  of  the  command  units  (WOC,  ATOC,  Army  G-3,  even  the  ASOC)  will 
typically  have  a  major  effect  on  the  models  of  those  units.  Due  to  the  fact  that  the  models  are  built  to 
perform  specific  functions  under  certain  conditions,  changing  those  functions  could  conceivably  require  a 
redesign  and  redevelopment  of  the  model.  At  the  very  least,  the  code  controlling  a  particular  function 
would  have>  to  change. 

In  summary,  then,  the  more  stable  the  environment  is,  the  less  it  will  cost  to  maintain  a  SuperKEATS 
system.  Every  system  that  works  in  the  real  world  where  changes  occur  will  require  maintenance  and 
upgrades.  There  is  no  evidence  that  a  simulation  practice  environment  will  necessarily  require  either  more 
or  less  maintenance  than  a  traditional  ITS. 


Prerequisites  for  Modeling 

Though  the  design  and  development  of  SuperKEATS  occurred  over  a  12-month  period,  important 
and  necessary  work  preceded  this  effort.  The  first  and  most  obvious  prerequisite  was  to  acquire 
knowledge  of  the  ASOC  domain.  Designing  SuperKEATS  required  knowing  what  people,  organizations, 
and  things  affected  ASOC  decision-making  during  a  battle  and  how  these  objects  interacted  with  the 
ASOC  and  among  themselves.  This  knowledge  acquisition  work  on  the  ASOC  domain  was  the  focus  of 
the  second  phase  of  this  project  (Brecke  et  al.,  1989)  and  occurred  over  a  period  of  about  1  year. 
Starting  with  interviews  of  ASOC  personnel  to  obtain  the  general  structure  of  the  ASOC  environment,  the 
acquisition  activity  then  proceeded  to  build  a  model  of  ASOC  decision-making  in  the  KEATS  system. 
Refinement  of  the  models  in  KEATS,  so  that  the  basis  for  ASOC  decision-making  (e.g.,  rules)  could  be 
obtained,  became  the  starting  point  for  the  models  in  SuperKEATS. 

As  the  design  of  SuperKEATS  proceeded,  it  became  clear  that,  for  realistic  modeling,  knowledge  of 
the  operations  of  other  decision-making  organizations  in  the  immediate  ASOC  environment  was 
important.  For  example,  the  model  of  a  WOC  had  to  respond  to  ASOC  tasking  requests.  But  to  know 
how  the  WOC  would  respond  in  every  circumstance  of  alert  status,  lost  aircraft,  and  previous  smooth  flow 
commitments  would  require  a  very  detailed  understanding  of  how  a  WOC  operates.  This  level  of  detail 
would  also  be  needed  for  the  ALO  submitting  air  requests  to  the  ASOC  and  assigning  smooth  flow 
missions  to  requests,  and  for  corps  commanders  changing  air  support  fire  priorities.  In  the  case  of 
SuperKEATS,  support  of  a  WOC  expert  was  requested  but  was  unobtainable.  The  knowledge  of  how 
these  decision-making  nodes  operated  came  either  from  what  the  ASOC  members  knew  and  expected  of 
a  WOC  or  otherwise  from  making  the  developer’s  best  guess. 
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Besides  the  models  requiring  extensive  detail,  assumptions  had  to  be  made  as  to  how  much  fidelity 
was  necessary  for  modeling  all  other  objects  in  the  environment.  In  the  case  of  SuperKEATS,  the 
simulation  limitations  (Table  3)  were  chosen  to  simplify  the  modeling  of  Army  units  so  that  only  actions 
important  to  the  ASOC  would  result. 

As  a  final  note  on  prerequisites,  it  is  suggested  that  having  an  in-house  expert  (e.g.,  a  retired  ASOC 
member)  available  during  the  building  of  SuperKEATS  would  have  resolved  many  “developer  best  guess” 
problems.  That  SuperKEATS  was  built  without  this  help,  and  was  readily  accepted  by  ASOC  personnel, 
shows  that  it  can  be  done  successfully  with  very  limited  access  to  the  experts.  However,  the  product  can 
be  no  better  than  the  knowledge  it  contains.  It  will  probably  require  one  or  more  iterations  of 
expert-use/model-revision  to  correct  developer  misconceptions.  Having  an  expert  always  available  would 
surely  have  speeded  or  eliminated  the  iterative  process. 

Transfer  to  Other  Domains 

The  modeling  in  SuperKEATS  should  be  easily  transferable  to  other  positions  in  the  ASOC  such  as 
G-3  Air,  Intel,  or  the  Director.  In  that  the  members  of  the  ASOC  function  as  a  team,  they  would  all 
receive  the  same  briefings  as  the  FDO  and  be  concerned  with  the  same  air  requests.  The  Intel  and  G-3 
Air  decisions  about  air  requests  are  not  concerned  with  tasking  specific  air  assets  but  are  accept/reject 
decisions  based  on  request  validity.  There  would  need  to  be  additional  modeling  so  that  spurious  requests 
were  generated  occasionally. 

The  ATOC  also  makes  decisions  about  supply  and  demand  of  aircraft;  in  certain  theaters  the  ATOC 
even  subsumes  the  ASOC  role  of  tasking  air  assets.  Hence,  the  decision-making  about  CAS  built  into 
SuperKEATS  should  be  directly  usable  by  ATOC  personnel  making  those  same  decisions.  If  it  was 
desired  to  task  Battlefield  Air  Interdiction  (BAI)  as  well  as  CAS,  then  the  modeling  would  need  to  be 
expanded. 

The  Army  modeling  in  SuperKEATS  is  designed  such  that  Army  units  request  support  when 
objectives  are  not  being  met.  Transfer  to  other  decision  environments  without  the  need  for  Army  requests 
would  require  different  models.  For  example,  a  node  for  air  defense  would  require  models  of  enemy  air 
units  launching  attacks,  and  models  for  results  of  air  encounters. 


Implementation  Lessons 


During  a  multiyear  project,  there  are  likely  to  be  changes  in  both  the  initial  concepts  (although  not  the 
goals)  and  the  available  tools.  The  original  concept  for  this  project  was  that  to  train  decision-making  in  a 
tactical  environment  requires  some  kind  of  an  intelligent  tutoring  system  (ITS) .  As  instructional  design 
theories  were  studied  and  as  results  from  existing  ITSs  became  available,  this  concept  changed  to  one  of 
developing,  as  the  product  of  this  effort,  a  simulation  practice  environment.  During  the  early  stages  of  this 
project,  the  need  for  a  fast  prototyping  environment  was  recognized  but  it  was  also  considered  that  a 
rule-based  expert  system  would  be  required  in  the  ITS.  Based  on  these  considerations  and  on  the  need  to 
gain  familiarity  with  the  software,  the  Knowledge  Engineering  Environment  (KEE)  was  purchased  during 
the  first  phase  of  the  project. 

During  the  second  phase  of  the  project,  when  knowledge  acquisition  was  the  area  of  activity,  the 
KEATS  system  was  built.  Because  KEATS  had  no  requirement  for  an  expert  system  and  but  did  have  the 
goal  to  place  the  system  physically  in  the  ASOC  for  their  use,  it  was  decided  to  use  the  object-oriented 
Smalltalk  V,  which  had  just  become  available  and  ran  on  personal  computer  (PC)  hardware.  KEATS  was 
developed  successfully  in  Smalltalk  V,  which  proved  to  be  an  excellent  environment  for  rapid  prototyping. 
The  only  drawbacks  were  that  response  time  was  somewhat  slow  on  AT  (80286-based)  machines  (but 
80386  machines  were  significantly  better)  and  the  medium-resolution  graphics  limited  the  amount  of 
information  that  could  be  presented  on  the  screen  at  one  time. 

When  the  third  phase  started,  it  was  recognized  that  using  software  already  developed  in  KEATS 
would  represent  a  savings  for  SuperKEATS.  However,  the  necessity  to  write  rules,  and  the  investment  in 
KEE  and  the  hardware  to  run  it,  led  to  developing  SuperKEATS  in  KEE.  Although  KEE  did  have  an 
object-oriented  environment,  a  nice  graphical  interface,  and  an  easy-to-use  knowledge  base  editor  with 
capabilities  for  multiple  inheritance,  it  was  not  well-suited  for  a  simulation  environment  for  these  reasons: 

1.  The  simulation  was  procedural  and  algorithmic  whereas  KEE  is  based  in  Lisp,  which  is  a 
symbolic  language.  Hence,  writing  the  algorithms  was  clumsy. 

2.  The  programming  support  tools,  such  as  the  editor  and  debugger,  for  the  Lisp  code  were  poor 
compared  to  those  for  Smalltalk  V. 

3.  Garbage  collection  (the  standard  means  of  recovering  unused  memory  with  the  Lisp  language) 
was  to  virtual  memory  on  disk  and  hence  very  slow  — typically  on  the  order  of  minutes.  This  is 
not  acceptable  in  a  real-time  interactive  environment. 
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4.  Memory  fragmentation  occurred  after  running  for  several  hours  of  an  exercise  and  this  slowed 
system  response  down  considerably.  Each  workstation  was  configured  with  16  MBytes  of  main 
memory,  but  for  the  final  evaluation,  8  MBytes  was  moved  from  one  system  to  the  “demo” 
system,  giving  it  24  MBytes  (the  workstation  limit).  Even  then,  later  in  the  exercise  the  expert 
found  system  response  to  be  slower  than  desired. 

5.  The  rule  system  proved  to  be  awkward  for  the  kind  of  rules  needed  for  suggestions  and 
feedback.  Many  rules  involved  comparisons  of  values  (number  of  tanks,  number  of  aircraft, 
etc.)  and  this  required  going  into  Lisp  code.  The  syntax  of  the  rules  probably  could  not  be  read 
and  understood  by  an  ASOC  expert. 

The  objected-oriented  approach  to  building  a  decision-making  environment  proved  to  be  very 
effective  in  SuperKEATS,  as  it  was  in  KEATS.  Despite  the  drawbacks  to  rule  implementation  in  KEE 
mentioned  above,  rules  were  a  simple  and  efficient  way  to  represent  ASOC  tasking  knowledge  within 
SuperKEATS.  Rules  typically  reason  about  the  current  state  of  specific  objects  and  are  easily  added  to 
the  object-oriented  simulation  environment. 


ASOC  Use  of  SuperKEATS 

From  the  data  presented  earlier  in  Section  III,  two  items  that  particularly  stand  out  are:  (a)  the  large 
difference  between  expert  and  novice  performance;  and  (b)  the  expressed  feeling  of  both  expert  and 
novice  that  suggestions  and  feedback  are  very  useful  for  training. 

That  SuperKEATS  is  clearly  responsive  to  differences  in  expertise  was  shown  by  comparing  the  results 
of  the  ASOC  expert  and  novice  on  the  complex  game.  This  is  illustrated  by  Figure  15  where  the  enemy 
has  penetrated  10  kilometers  into  Blue  territory  in  2  hours  with  novice  support,  whereas  in  4  hours  with 
expert  support,  the  enemy  has  been  held  to  within  1  kilometer  of  Blue  territory.  Though  the  sample  size 
is  too  small  to  generalize,  it  should  be  noted  that  both  ASOC  members  who  used  SuperKEATS  were  on 
active  duty  and  would  be  in  combat  if  the  need  arose.  If,  in  an  emergency,  the  novice  were  thrust  into  the 
position  of  ASOC  decision-maker,  the  results  here  indicate  that  the  air  support  effectiveness  would  be 
much  lower  than  with  a  well-trained  expert  and  this  condition  might  have  a  deleterious  effect  on  the 
course  of  a  battle. 

Although  the  effectiveness  of  the  suggestions  and  feedback  have  not  been  tested,  the  enthusiasm  of 
the  ASOC  personnel  for  this  aid  to  training  is  noted.  The  novice  thought  that  the  feedback  was  the  best 
feature  of  SuperKEATS  and  suggested  even  more  extensive  suggestion  and  feedback  information.  The 
expert  thought  tnat  feedback  was  important  for  the  training  of  novices  and  for  the  practice  of  experts.  He 
also  thought  suggestions  would  be  useful  for  real-world  (i.e.,  combat)  tasking. 
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Figure  IS.  Results  of  Expert  vs  Novice  Tactical  Decision-Making. 
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Conclusions 


It  has  been  demonstrated  that  it  is  feasible  to  build  a  single-user  simulation  environment  for  practicing 
tactical  decision-making  skills  that  runs  on  a  microprocessor-based  computer.  It  is  noted  that  the  current 
prototype  can  also  function  as  a  vehicle  for  training  research  to  find  optimum  strategies  for  decision 
training.  The  effectiveness  of  training  with  the  prototype  has  not  been  tested  nor  has  any  training  research 
been  done  at  this  time.  The  initial  evaluation  by  users  from  the  operational  community  was  that  the 
prototype  would  be  useful  for  their  training. 

It  is  recommended,  when  building  a  simulation  practice  environment,  that  the  simulation  be  built  first 
using  an  object-oriented  approach  with  instructional  features  added  later.  Developing  a  credible 
simulation  that  is  easy-to-use  and  that  motivates  the  user  was  the  major  technical  challenge  of  this  work. 
Once  the  simulation  was  operational,  it  was  a  straightforward  process  to  add  instructional  features  based 
on  domain  knowledge  represented  as  rules. 

It  is  recommended  that  if  a  knowledge  engineering  tool  (e.g.,  KEATS)  is  built  first,  then  the 
simulation  practice  environment  (e.g.,  SuperKEATS)  that  follows  should  be  built  using  software 
developed  in  the  precursor.  If  another  simulation  practice  environment  is  built  or  if  SuperKEATS  is  to  be 
rehosted  to  another  machine,  it  is  not  recommended  that  it  be  developed  in  KEE,  with  its  problems  of 
Lisp  garbage  collection  to  virtual  memory  on  disk  and  memory  fragmentation  slowing  system  response 
times. 

A  key  prerequisite  for  building  a  simulation  practice  environment  is  having  good  access  to  experts. 
The  ideal  situation  for  both  knowledge  acquisition  and  training  system  development  is  to  have  the  expert 
in-house.  Other  factors  to  consider  in  building  a  simulation  practice  environment  are  the  stability  of  the 
environment  over  the  time  it  takes  to  build  the  system  and  how  general,  and  hence  transferable  to  other 
command  and  control  nodes,  is  the  environment. 


48 


V.  FUTURE  DIRECTIONS 


The  main  objective  of  this  project  phase  was  to  build  a  prototype  simulation  environment  that  allows 
the  user  to  practice  decision-making  skills  and  permits  experimentation  with  instructional  strategies. 
Building  that  prototype,  SuperKEATS,  has  been  the  main  topic  of  this  report.  SuperKEATS  meets  its 
goals  in  providing  a  reactive  environment  to  make  decisions.  Although  the  ASOC  members  indicated  that 
SuperKEATS  should  be  useful  to  the  ASOC,  it  remains  to  be  shown  that  it  provides  effective  training  for 
decision-making.  Only  resource  constraints  prevented  this  more  complete  testing  and  evaluation  from 
being  done  during  this  project  phase.  It  is  recommended  that  SuperKEATS  be  tested,  after  minor 
modifications  suggested  by  the  ASOC  expert,  to  see  how  effectively  the  embedded  decision-making 
knowledge  is  learned.  This  testing  should  be  thorough  and  complete,  using  a  statistically  sufficient  number 
of  novice  students,  and  include  a  control  group  that  does  not  use  SuperKEATS. 

Adding  more  ASOC  decision-making  knowledge  to  SuperKEATS  before  testing  should  also  be 
considered.  Knowledge  acquisition  using  KEATS  was  very  abbreviated  during  this  project  and  we  are 
concerned  that  the  knowledge  currently  embedded  in  SuperKEATS  may  not  provide  the  breadth  of 
training  that  would  be  needed  in  combat.  Knowledge  acquisition  with  KEATS  could  be  extended, 
particularly  using  exercises  built  by  ASOC  experts.  The  resulting  rules  could  then  be  incorporated  into 
SuperKEATS  and  form  the  basis  for  evaluation  of  training  effectiveness. 

Any  new,  prototype  system  offers  many  opportunities  for  improvement,  and  SuperKEATS  is  no 
exception.  During  the  ASOC  evaluation,  numerous  suggestions  were  obtained  that  would  make  an 
easier-to-use  system  and  provide  additional  information.  In  addition,  extensions  beyond  the  current 
capabilities  include:  (a)  provide  multiple  screens  to  display  the  large  amount  of  data  that  must  be  accessed 
for  decision-making;  (b)  make  available  to  an  exercise  builder  certain  simulation  parameters  that  are  now 
in  code,  such  as  time  between  fire  priority  updates,  chance  of  base  closures,  average  length  of  base 
closures,  etc.;  (c)  add  a  weather  model  so  that  weather  becomes  a  factor  in  decisions;  and  (d)  use  real 
maps  and  add  simple  terrain  rules  for  Army  movement. 

The  original  concept  was  to  field  a  decision  training  system  on  a  microcomputer  that  would  be 
available  at  the  operation  centers  themselves.  Because  SuperKEATS  was  developed  on  a  workstation  with 
Lisp-based,  expensive  KEE  software  that  has  heavy  requirements  for  memory  and  disk  space,  a  possible 
direction  would  be  to  port  it  to  the  personal  computer  environment  using  an  object-oriented  language 
such  as  Smalltalk  V,  in  which  KEATS  was  developed.  The  disadvantages  to  such  a  machine  would  be  that 
the  lower-resolution  screen  would  not  support  windowing,  as  on  a  workstation,  and  it  would  also  require  a 
considerable  amount  of  memory. 
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SuperKEATS  could  also  be  extended  to  other  environments  of  decision-making  both  within  the 
ASOC,  as  at  the  G-3  Air  and  Intel  positions,  and  at  nodes  that  are  involved  with  supply  and  demand  air 
support  decisions  such  as  at  an  ATOC.  It  is  recommended  that  the  approaches  and  technology  developed 
here  be  applied  to  an  ATOC.  Starting  from  the  beginning,  knowledge  engineering  should  begin  with 
interviews  to  scope  the  domain.  A  KEATS-like  system,  with  scripted  events,  would  be  developed  to  do 
detailed  knowledge  acquisition  and  provide  a  training  quid  pro  quo.  SuperKEATS  would  then  evolve  on 
the  same  hardware/software  platform  as  KEATS,  providing  the  simulation  practice  environment  and 
retaining  the  knowledge  engineering  capabilities  of  KEATS.  This  would  provide  the  capability  to  maintain 
and  upgrade  SuperKEATS  by  doing  knowledge  engineering  as  the  environment  changes  and  immediately 
incorporating  the  new  knowledge  into  the  system. 
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ABBREVIATIONS  AND  ACRONYMS 


ALO 

Air  Liaison  Officer 

ASOC 

Air  Support  Operations  Center 

ASW 

Anti-Submarine  Warfare 

ATOC 

Allied  Tactical  Operations  Center 

C2 

Command  and  Control 

CAS 

Close  Air  Support 

CDT 

Component  Display  Theory 

FDO 

Fighter  Duty  Officer 

ISA 

Initial  Situation  Assessment 

ISD 

Instructional  Systems  Design 

ITS 

Intelligent  Tutoring  System 

KEATS 

Knowledge  Engineering  and  Training  System 

KEE 

Knowledge  Engineering  Environment 

POA 

Plan  of  Action 

UCSD 

University  of  California  at  San  Diego 

woe 

Wing  Operations  Center 
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APPENDIX  A 


SuperKEATS  Knowledge  Bases 


SuperKEATS  knowledge  bases  contain  objects,  called  units  in  KEE.  Functionally  equivalent  to  an  object, 
frame,  or  schema  in  other  object-oriented  programming  environments,  a  KEE  unit  represents  the  definable 
attributes  and  behaviors  of  a  physical  or  conceptual  object.  Units  which  share  the  same  attributes  and 
behaviors  are  grouped  into  classes.  All  SuperKEATS  knowledge  bases  and  their  major  classes  and  units  follow. 
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AGENDA  KB 

Contains  all  event  objects;  initiates  timed  events  and 
keeps  track  of  pending  events. 

AGENDA-MANAGER 

DATE-TIME 

EVENT 

ADD-ALERT-EVENT 

ADD-SMOOTHFLOW-EVENT 

ADJUST-DOWN-TIME-EVENT 

AIR-REQUEST-EVENT 

AIRCRAFT-AT-X-MINUTE-READINESS-EVENT 

AIRCRAFT-GET-READY-EVENT 

AIRCRAFT-ON-5-MINUTE-ALERT-EVENT 

AIRCRAFT-RETURNS-HOME-EVENT 

AIRCRAFT-SCHEDULED-FOR-ALERT-EVENT 

CANCEL-SMOOTHFLOW-EVENT 

CHANGE-RESERVE-SUPPORT-EVENT 

IN-FLIGHT-REPORT-EVENT 

MISSION-OVER-CONTACT-POINT-EVENT 

MISSION-OVER-TARGET-EVENT 

MISSION-RECOVERY-EVENT 

MISSION-TAKE-OFF-EVENT 

SIMULATION-UPDATE-EVENT 

TANKS-HAVE-ARRIVED-EVENT 

A1R-FQBCE  KB 

Contains  classes  for  air  force  objects  and  their 
behavior. 

AIR-FORCE-ADMIN 

AF-GAME-INITIAUZER 
WOC-RECOVERY-SCRATCHPAD 
WOC-SCRATCHPAD 
AIR-FORCE-CONSTRAINTS 
WOC-CONSTRAINT -CHECKER 
AIR-FORCE-UNIT 
AIRCRAFT 
SQUADRONS 

woes 

BASES 

MISSIONS 

SMOOTHFLOW-DONE-CONDiTIONS 

ENOUGH-MISSIONS-DONE-CONDITION 
LAST  -NLT -DONE-CONDITION 
NO-SORTIES-DONE-CONDITION 
WOC-CONSTRAINTS 

CANCEL- ALERT  -CONSTRAINTS 
ALERT-CANCELLED-CONSTRAINT 
CHANGE-ALERT-GROSS-CONSTRAINTS 
ALERT-CANCELLED-CONSTRAINT 
BASE-NOT  -OPERATION  AL-FOR-ALERT 
CONSTRAINT 


BASE-NOT-REOPENED-FOR-ALERT- 

CONSTRAINT 

CHANGE-ALERT-EXCEEDS- 

MAXIMUM-CONSTRAINT 

CANCEL-MISSION-CONSTRAINTS 

CANCEL-TOO-LATE-CONSTRAINT 
MISSION-CANCELLED-CONSTRAINT 
CANCEL-SMOOTHFLOW-CONSTRAINTS 
CANCEL-SMOOTHFLOW-MISSION- 
TOO-  LATE-CONSTRAINT 
C  ANCE  L-SMOOTHFLOW-MISSION- 
UNKNOWN-CONSTRAINT 
CANCEL-SMOOTHFLOW-TOO-LATE- 
CONSTRAINT 

SMOOTHFLOW-CANCELLED-CONSTRAINT 

CHANGE-ALERT-AVAIL-CONSTRAINTS 

CHANGE-ALERT-PRIOR-COMMITMENTS- 

CONSTRAINT 

CHANGE-ALERT-AVAIL-QUAL-CONSTRAINTS 
CH  ANGE-ALERT  -EXCEEDS-M  AXIMUM- 
SHORT  AGE-CONSTRAINT 
CHANGE-ALERT-GROSS-CONSTRAINTS 

CHANGE-ALERT-INVALID-CONSTRAINT 

CHANGE-ALERT-NO-SORTIES-CONSTRAINT 

CHANGE-ALERT-NUMBER-UNAVAILABLE- 

CONSTRAINT 

CHANGE-ALERT-QUAL-CONSTRAINTS 

CHANGE-ALERT-QUALIFIED-BASE- 

CONSTRAINT 

CHANGE-ALERT-QUALIFIED-PILE- 

CONSTRAINT 

CHANGE-ALERT-QUALIFIED-SQUADRON- 

CONSTRAINT 

DIVERT-MISSION-CONSTRAINTS 

BASE-NOT-REOPENED-FOR- 

REDIRECTED-TAKEOFF- 

CONSTRAINT 

DIVERT-TOO-LATE-CONSTRAINT 

INTERFERES-WITH-COMMITMENTS- 

CONSTRAINT 

MISSION-CANCELLED-CONSTRAINT 
WONT-M  AKE-NLT -CONSTRAINT 
DIVERT-SMOOTHFLOW-CONSTRAINTS 
DIVERT -SMOOTHFLOW-MISSION- 
TOO-LATE-CONSTRAINT 
DIVERT -SMOOTHFLOW-MISSION- 
UNKNOWN-CONSTRAINT 
DIVERT -SMOOTHFLOW-TOO-LATE- 
CONSTRAINT 

SMOOTHFLOW-CANCELLED-CONSTRAINT 

FLOW-AVAIL-CONSTRAINTS 

CANT  -M  AKE-NLT -CONSTRAINT 
FUDGED-PRIOR-COMMITMENTS- 
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CONSTRAINT 

NOT-ENOUGH-MISSIONS-CONSTRAINT 

FLOW-GROSS-CONSTRAINTS 

BASE-NOT-REOPENED-FOR-TAKEOFF- 

CONSTRAINT 

BEFORE-FIRST-LIGHT-CONSTRAINT 

BELOW-THREAT-MIN-CONSTRAINT 

FUDGED-NO-SORTIES-CONSTRAINT 

FUDGED-NUMBER-UNAVAILABLE- 

CONSTRAINT 

LAST-LIGHT-CONSTRAINT 

NLT-BEFORE-TOT-CONSTRAINT 

NO-JAAT-CONSTRAINT 

FLOW-QUAL-CONSTRAINTS 

QUALIFIED-FOR-JAAT-CONSTRAINT 
QUALIFIED-T ASKING-BASE-CONSTRAINT 
QUALIFIED-T  ASKING-PILE-CONSTRAINT 
PUT-ON-ALERT-AVAIL-CONSTRAINTS 
PRIOR-COMMITMENTS-CONSTRAINTS 
PUT -ON-ALERT  -AVAIL-QUAL-CONSTR  AINTS 
EXCEEDS-M  AXIMUM-SHORT  AGE- 
CONSTRAINT 

PUT-ON-ALERT-GROSS-CONSTRAINTS 

ALERT-TOO-EARLY-CONSTRAINT 

ALERT-TOO-LATE-CONSTRAINT 

ALL-SQUADRONS-ALERT-CONSTRAINT 

BASE-NOT-OPERATIONAL-FOR- 

ALERT-CONSTRAINT 

BASE-NOT-REOPENED-FOR-ALERT- 

CONSTRAINT 

EXCEEDS-MAXIMUM-CONSTRAINT 
NO-SORTIES-CONSTRAINT 
NUMBER-UNAVAILABLE-CONSTRAINT 
PAST-ALERT-TIME-CONSTRAINT 
SQUADRON- ASSIGNED-ALERT  - 
CONSTRAINT 

PUT-ON-ALERT-QUAL-CONSTRAINTS 

QUALIFIED-ALERT-BASE-CONSTRAINT 

QUALIFIED-ALERT-PILE-CONSTRAINT 

QUALIFIED-ALERT-SQUADRON- 

CONSTRAINT 

T  ASKING-AVAIL-CONSTRAINTS 
CANT-M  AKE-NLT -CONSTRAINT 
FUDGED-PRIOR-COMMITMENTS- 
CONSTRAINT 

TASKING-GROSS-CONSTRAINTS 

BASE-NOT-REOPENED-FOR-TAKEOFF- 

CONSTRAINT 

BEFORE-FIRST-LIGHT-CONSTRAINT 

BELOW-THREAT-MIN-CONSTRAINT 

FUDGED-NO-SORTIES-CONSTRAINT 

FUDGED-NUMBER-UNAVAILABLE- 

CONSTRAINT 


LAST-LIGHT-CONSTRAINT 

NO-JAAT-CONSTRAINT 

TASKING-QUAL-CONSTRAINTS 

QUALIFIED-FOR-JAAT-CONSTRAINT 

QUALIFIED-TASKING-BASE-CONSTRAINT 

QUALIFIED-TASKING-PILE-CONSTRAINT 


ANSWER  KB 

Contains  objects  used  to  format  and  display 
information  accessed  via  the  ASK  button. 

ANSWER-MAN 

INIT 

SQUADRON-WINDOW 

WINDOW 


ABM.Y-KB 

Contains  classes  for  army  objects  and  their 

behavior. 

ALO 

ARMY-UNIT 

COMMAND-UNIT 

CORPS 

BLUE-CORPS 

RED-CORPS 

FRONT-LINE-DIVISION 

BLUE-FRONT-LINE-DIVISION 
RED-FRONT-LINE-DIVISION 
FRONT -LINE-T ANK-B  ATT  ALION 
FRONT-LINE-TANK-BRIGADE 
FIGHTING-UNIT 

FRONT-LINE-TANK-COMPANY 

BLUE-FRONT-LINE-TANK-COMPANY 

RED-FRONT-LINE-TANK-COMPANY 

RESERVE-UNIT 

RESERVE-DIVISION 

RESERVE-TANK-BATTALION 

RESERVE-TANK-BRIGADE 

RESERVE-TANK-COMPANY 

COMPLEX  KB 

Contains  an  object  which  describes  the  game  with 

the  highest  complexity  level. 

COMPLEX 


57 


FACTS  KB 


Contains  classes  for  aircraft  types,  ordnances,  and 
SCLs  which  are  used  in  the  display  of  factual 
information  accessed  via  the  INFO  button. 

AIRCRAFT-TYPES 

LOAD-COMPONENTS 

GUNS 

ORDNANCE 

BOMBS 

CBUS 

ANTI-ARMOUR-CBUS 

GENERAL-PURPOSE-CBUS 

MISSILES 

SCLS 

GAME-CONTROL  KB 

Contains  objects  which  describe  the  currently 
selected  game  and  objects  used  in  the 
management  of  game  "mode”  changes  (game 
selection,  plan  and  assessment,  etc.). 

CB- WINDOW 

ERR-WINDOW 

GAMES 

GAMES-MANAGER 

INIT 

INITIAL-WINDOW 
INSTRUCTIONS-INFO- WINDOW 
INSTRUCTIONS-TEXT 
LOAD-GAME-FORM 
PLANNING-SCRATCHPAD-CONTROL 
SUM- WINDOW 

GAME-STUFF  KB 

Contains  classes  and  instances  of  objects  created 
dynamically  during  a  game. 

AIR-REQUEST 

ALERT-REQUEST 

ASK-TELL-WOC 

ASOC-TELLS-WOC 

RULES-ASK-WOC 

RULES-TASKING-REQUEST 

GAME-STATE 

PREPLANNED-REQUEST 

SMOOTHFLOW-REQUEST 

WOC-RESPONSES 

OTHER-RESPONSE  PUT-ON-ALERT- 
RESPONSE 

TASKING-RESPONSE 


GAME-SUM  KB 

Contains  objects  used  to  create  and  display  game 
summary  displays  (text,  stripcharts,  bar  graphs, 
etc.) 

GAME-SUM-VALUES 

GC-SUM-STRIPCHARTS 

GC-SUM-VERTICAL-BAR-GRAPH-ACTUATORS 

INFO  KB 

Contains  objects  used  to  format  and  display 
briefings,  orders,  and  factual  information  which  is 
accessed  via  the  INFO  button. 

FACTS-WINDOW 

FDO-SHIFT-CHANGE-WINDOW 

G3-AIR-WINDOW 

INFO-MAN 

INIT 

INTEL-WINDOW 

ORDERS-WINDOW 

1N.IERF,  KB 

Contains  objects  defining  the  Top  Bar  buttons  and 
the  message  handler  between  the  ASOC  and  the 
army /woes. 

ASOC-INPUT-HANDLER 

BUTTON 

MENU-BUTTONS 

AIR-REQUEST-BUTTON 

ASK-BUTTON 

GAME-CONTROL-BUTTON 

INFO-BUTTON 

MSG-LOG-BUTTON 

STATUS-BOARD-BUTTON 

TELL-BUTTON 

METHOD-BUTTONS 

MAP-BUTTON 

NEXT-BUTTON 

INIT 

INTERF-GAME-INIT 
INTERMEDIATE  KB 

Contains  an  object  which  describes  the  game  with 
an  intermediate  level  of  complexity. 

INTERMEDIATE 
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MAP  KB 

Contains  objects  to  display  positional  information 
for  air  requests,  bases,  corps,  and  divisions  on  a 
grid. 

MAP-MAKER 
MSG-LOG  KB 

Contains  objects  used  to  format  and  display  WOC 
responses  to  actions  initiated  by  the  user;  also 
handles  the  display  of  air  request  information. 

AIR-REQUEST-MAN 

FEEDBACK-WINDOW 

INIT 

MSG-LOG-MAN 
MSG- WINDOW 
REQUEST-WINDOW 
TEXT-BOX 

SIMPLE  KB 

Contains  an  object  which  describes  the  game  with 
the  lowest  level  of  complexity. 

SIMPLE 

SIM-UPDATE  KB 

Performs  simulation  updates  at  game-specified 
intervals. 

INIT 

SIMULATION-UPDATE 

SK5ATS-KB 

Handles  initialization  of  other  KBs  on  start-up. 
SKEATS-MANAGER 

SOME-RULES  KB 

Contains  rules  for  tasking  and  smoothflow 
suggestions. 

END-OF-GAME-CLEAN-UP 

SMOOTHFLOW-INFO 

SMOOTHFLOW-RULES 

DETERMINING-SMOOTHFLOW-NEED 
CHECK-REQUEST-TYPE 
CHECK-TIME-CONSTRAINT  1 
COUNT-SMOOTHFLOW-TYPE-REQUESTS 
DIVISION-NEEDS-SMOOTHFLOW 
DETERMINING-SMOOTHFLOW-RESOURCES 
SELECTION-OF-SQN-FOR-SMOOTHFLOW 


SOME-RULES-INFO 

TASKING-RULES 

DIVERT-RULES 

DIVERT-SMOOTHFLOW 

ORDER-OF-TASKING 

REQUESTS-FOR-DIFFERENT-DIVISIONS 

SAME-TIME-REQUESTS 

SERVICE-IN-CONTACT-FIRST 

SERVICE-PRIORITY-FIRST 

REQUEST-REFUSAL 

REFUSE-DUE-TO-SHORT-TIME 
SE  LECTION-OF- ASSETS 

AMOUNT-OF-SUPPORT 

MANY-TANKS 

THREAT-PROBLEMS 

CHOOSE-BEST-AGAINST-THREAT 

USE-NOT-ON-ALERT-AC 

NOT-ASAP-WITH-TIME 

STATUS-BOARD  KB 

Contains  objects  used  to  format  and  display  data 
for  the  Summary  Status  Board,  the  Mission  Status 
Boards  and  the  Base  Summary  Board  (formerly 
known  as  the  Intel  Board) . 

IN-WINDOW 

INIT 

MB- WINDOW 
SB-WINDOW 
STATUS-BOARD-MAN 

T.51U  KB  ANP  TELL2  KB 

Contains  objects  which  provide  the  means  for  the 
user  to  interface  with  a  WOC  or  an  ALO  (e.g.,  to 
TELL  the  WOC  to  put  aircraft  on  alert) . 

TELL-COMMUNIQUES 

TELL-CANCEL-ALERT 

TELL-CANCEL-MISSION 

TELL-CANCEL-SMOOTHFLOW 

TELL-CHANGE-ALERT 

TELL-DIVERT-MISSION 

TELL-DIVERT-SMOOTHFLOW 

TELL-NEED-SEAD 

TELL-PUT-ON-ALERT 

TELL-REFUSE-TO-TASK 

TELL-TASK-MISSIONS 

TELL-SMOOTHFLOW-MISSIONS 

TELL-MULTIPLE-MISSIONS 

TELL-MAN 

TELL-WINDOW-MANAGER 

TELL-COMMUNIQUES-PART2 

TELL-CHANGE-ALERT 

TELL-ERROR 
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APPENDIX  B 


Protocol  of  the  Expert  Playing  the  SuperKEATS  Complex  Game 
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210000  :  UOC  TFW1  ACCEPTED  request  to  put  12  AlOs  on  alert  at  210645. 


210000  :  UOC  TFU1  REFUSED  request  to  put  12  AlOs  on  alert  at  210645. 

REASON:  •  NOT  ENOUGH  FACILITIES  AT  BASE  FOR  REOUESTED  NUMBER  OF  ALERTS 
(See  Base  Summary  Board  for  maximum  alert  allocations.) 

210000  :  UOC  TFU1  ACCEPTED  request  to  put  8  AlOs  on  alert  at  210645. 

210000  :  UOC  TFU3  ACCEPTED  request  to  put  20  A7s  on  alert  at  210700. 

210000  :  UOC  TFU9  ACCEPTED  request  to  put  20  A JETS  on  alert  at  210645. 

210000  :  UOC  TFU6  ACCEPTED  request  to  put  20  F5s  on  alert  at  210700. 

210000  :  UOC  TFU10  ACCEPTED  request  to  put  20  HARs  on  alert  at  210700. 

210000  :  The  following  mandatory  information  is  missing  from  the  request: 

UOC/SQUADRON 

210000  :  UOC  TFU2  ACCEPTED  request  to  put  16  F4s  on  alert  at  210645. 

210445  :  TIME  UPDATE 
210500  :  TIME  UPDATE 
210505  :  TIME  UPDATE 
21Cj IS  :  TIME  UPDATE 
210520  :  TIME  UPDATE 
210530  :  TIME  UPDATE 
210545  :  TIME  UPDATE 
210645  :  TIME  UPDATE 
210700  :  TIME  UPDATE 

210700  :  BASE-3  has  been  closed  by  a  ground  attack.  Estimated  reopen  time:  210730. 

(For  details  on  mission  cancellations,  etc.  see  the  Mission  Status  Boards.) 

210700  :  GROUND  attack  on  BASE-3  has  resulted  in  the  loss  of  1  a/c  from  squadron  TFS32. 

210700  :  BASE- 10  has  been  closed  by  a  ground  attack.  Estimated  reopen  time:  210730. 

(For  details  on  mission  cancellations,  etc.  see  the  Mission  Status  Boards.) 

210700  :  TFWlO  will  assign  only  6  of  the  8  planned  a/c  to  30M  alert  until  more  become  available. 

210700  :  GROUND  attack  on  BASE- 10  has  resulted  in  the  loss  of  6  a/c  from  squadron  TFS101. 

210711  :  TIME  UPDATE 

210711  :  Received  ASAP  request  #3AC201  from  BLUE-DIV2-ALO  to  neutr  15  tanks,  NLT:  210900. 
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210711  :  Received  ASAP  request  K3AC301  from  BLUE-DIV3-AL0  to  neutr  14  tanks,  NLT:  210810. 


210711 

210711 

210711 

210711 

210711 

210711 

210715 

210716 

210716 

210716 

210716 

210716 

210716 

210716 

210719 

210719 

210720 

210720 

210721 

210721 

210725 

210725 

210726 

210726 


TFU6  will  assign  only  4  of  the  8  planned  a/c  to  30M  alert  until  more  become  available. 

UOC  TFU6  ACCEPTED  request  to  task  multiple  missions  for  request  83AC201. 

#3AC201  -  4  FSs  from  BASE-6 
#3AC201A  •  4  F5s  from  BASE-6 

TFU2  will  assign  only  2  of  the  4  planned  a/c  to  30M  alert  until  more  become  available. 

UOC  TFW2  ACCEPTED  request  83AC301  8  tasked  the  following  mission: 

03AC3O1  -  4  F4s  from  BASE -2 

BIUE-DIV2-ALO  reports  that  SEAD  is  coordinated  for  current  &  future  air  support. 

BLUE-DIV3-ALO  reports  that  SEAD  is  coordinated  for  current  &  future  air  support. 

TIME  UPDATE 

TIME  UPDATE 

Received  ASAP  request  #3AC302  from  BLUE-DIV3-ALO  to  neutr  15  tanks,  NLT:  210815. 
Mission  #3AC201  in  support  of  request  (P3AC201  is  TAKING  OFF. 

Mission  #3AC301  in  support  of  request  #3AC301  is  TAKING  OFF. 

TFU2  will  assign  only  6  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 

TFU2  will  assign  none  of  the  4  planned  a/c  to  30M  alert  until  more  become  available. 

UOC  TFU2  ACCEPTED  request  #3AC302  &  tasked  the  following  mission: 

#3AC302  •  4  F4s  from  BASE-2 

TIME  UPDATE 

Received  ASAP  request  #3AC101  from  BLUE-DIV1 -ALO  to  neutr  39  tanks,  NLT:  210820. 

TFU1  will  assign  only  4  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 

UOC  TFU1  ACCEPTED  request  83AC101  &  tasked  the  following  mission: 

83AC101  -  4  AlOs  from  BASE-1 

TIME  UPDATE 

UOC  TFU1  ACCEPTED  request  83AC101  &  tasked  the  following  mission: 

83AC101A  -  4  AlOs  from  BASE-1 

TIME  UPDATE 

Mission  83AC101  in  support  of  request  #3AC101  is  TAKING  OFF. 

TIME  UPDATE 

Mission  83AC302  in  support  of  request  03AC3O2  is  TAKING  OFF. 
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210728  :  TIME  UPDATE 

210728  :  Received  ASAP  request  #3AC202  from  BLUE-D1V2-ALO  to  neutr  14  tanks,  NIT:  210930. 
210729  :  TIME  UPDATE 

210729  :  Received  ASAP  request  «AC303  from  81UE-DIV3-ALO  to  neutr  12  tanks,  NLT :  210830. 

210730  :  TIME  UPDATE 

210730  :  BASE -3  has  reopened. 

210730  :  TFU6  will  assign  only  4  of  the  8  planned  a/c  to  15M  alert  until  more  become  available 

210730  :  TFU6  will  assign  none  of  the  8  planned  a/c  to  30M  alert  until  more  become  available. 

210730  :  WOC  TFW6  ACCEPTED  request  #3AC202  &  tasked  the  following  missions: 

#3AC202  -  4  F5s  from  BASE-6 
(K3AC202A  -  4  F5s  from  BASE-6 

210730  :  BASE- 10  has  reopened. 

210731  :  TIME  UPDATE 

210731  :  Mission  #3AC201A  in  support  of  request  #3AC201  is  TAKING  OFF. 

210732  :  TFU2  will  assign  only  2  of  the  4  planned  a/c  to  5M  alert  until  more  become  available. 

210732  :  TFU2  will  assign  none  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 

210732  :  TFW2  will  assign  none  of  the  4  planned  a/c  to  30M  alert  until  more  become  available. 

210732  :  WOC  TFU2  ACCEPTED  request  #3AC303  &  tasked  the  following  missions: 

(H3AC303  -  4  F4s  from  BASE -2 
#3AC303A  -  4  F4s  from  BASE-2 

210735  :  TIME  UPDATE 

210735  :  Received  ASAP  request  (K3AC304  from  BLUE-D1 V3-AL0  to  neutr  13  tanks,  NIT:  210835. 
210735  :  Mission  #3AC202  in  support  of  request  (K3AC202  is  TAKING  OFF. 

210737  :  TIME  UPDATE 

210737  :  Mission  03AC3O3  in  support  of  request  <f3AC303  is  TAKING  OFF. 

210740  :  TIME  UPDATE 
210741  :  TIME  UP0ATE 
210742  :  TIME  UP0ATE 

210742  :  Mission  #3AC201  in  support  of  request  #3AC201  is  RETURNING. 

TOT:  210737,  3  tanks  neutr,  no  a/c  tost. 

210743  :  TFU9  will  assign  only  2  of  the  6  planned  a/c  to  30M  alert  until  more  become  available 
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210743 

210745 

210745 

210747 

210747 

210747 

210748 

210748 

210751 

210751 

210751 

210751 

210751 

210751 

210752 

210753 

210755 

210755 

210756 

210756 

210757 

210757 

210757  : 

210757  : 

210757  : 


:  WOC  TFW9  ACCEPTED  request  IV3AC304  1  tasked  the  following  mission: 

#3AC304  •  4  AJETs  from  BASE-9 

:  TINE  UPDATE 

:  Mission  43AC202A  in  support  of  request  13AC202  is  TAKING  OFF. 

:  TIME  UPDATE 

:  Mission  #3AC303A  in  support  of  request  #3AC303  is  TAKING  OFF. 

:  Mission  N3AC301  in  support  of  request  N3AC301  is  RETURNING. 

TOT:  210742,  2  tanks  neutr,  no  a/c  lost. 

:  TIME  UPDATE 

Mission  #3AC304  in  support  of  request  #3AC304  is  TAKING  OFF. 

:  TIME  UPDATE 

:  Received  ASAP  request  N3AC102  from  BLUE-DIV1 -ALO  to  neutr  43  tanks,  NLT:  210850. 

Mission  #3AC101A  in  support  of  request  B3AC101  is  TAKING  OFF. 

:  TFU1  will  assign  none  of  the  4  planned  a/c  to  5M  alert  until  more  become  available. 

TFWl  will  assign  none  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 

WOC  TFWl  ACCEPTED  request  #3AC102  &  tasked  the  following  missions: 

#3AC102  -  4  AlOs  from  BASE-1 
X3AC102A  •  4  AlOs  from  BASE-1 

:  TIME  UPDATE 
:  TIME  UPDATE 
:  TIME  UPDATE 

Received  ASAP  request  B3AC203  from  BLUE-DIV2-AL0  to  neutr  18  tanks,  NLT:  210955. 

TIME  UPDATE 

Mission  B3AC102  in  support  of  request  #3AC102  is  TAKING  OFF. 

TIME  UPDATE 

TFU10  will  assign  only  6  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 

TFW10  will  assign  none  of  the  8  planned  a/c  to  30M  alert  until  more  become  available. 

WOC  TFW10  ACCEPTED  request  H3AC203  t  tasked  the  following  missions: 

B3AC203  -  4  HARs  from  BASE -10 
«3AC203A  -  4  HARs  from  BASE -10 


Mission  #3AC302  in  support  of  request  B3AC302  is  RETURNING. 
TOT:  210752,  2  tanks  neutr,  no  a/c  lost. 
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210757  :  Mission  #3AC201A  in  support  of  request  #3AC201  is  RETURNING. 

TOT:  210752,  5  tanks  neutr,  no  a/c  lost. 

210758  :  TIME  UPDATE 

210758  :  Mission  #3AC201  recovered  at  BASE-6. 

210758  :  TFU6  will  assign  none  of  the  8  planned  a/c  to  30M  alert  until  more  become  available. 
210759  :  TIME  UPDATE 
210800  :  TIME  UPDATE 

210800  :  WOC  TFW3  switching  alert  aircraft  for  squadron  TFS31. 

210802  :  TIME  UPDATE 

210802  :  Mission  #3AC101  in  support  of  request  #3AC101  is  RETURNING. 

TOT:  210752,  32  tanks  neutr,  1  a/c  lost. 

210802  :  Mission  N3AC203  in  support  of  request  N3AC203  is  TAKING  OFF. 

210803  :  TIME  UPDATE 

210804  :  TIME  UPDATE 

210804  :  Mission  #3AC202  in  support  of  request  #3AC2Q2  is  RETURNING. 

TOT:  210759,  5  tanks  neutr,  no  a/c  lost. 

210806  :  TIME  UP0ATE  ' 

210806  :  Mission  #3AC102A  in  support  of  request  *3AC102  is  TAKING  OFF. 

210807  :  TIME  UPDATE 

210808  :  TIME  UPDATE 

210808  :  Mission  13AC301  recovered  at  BASE-2. 

210808  :  TFU2  will  assign  only  2  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 

210808  :  TFW2  will  assign  none  of  the  4  planned  a/c  to  30M  alert  until  more  become  available. 

210808  :  Mission  03AC3O3  in  support  of  request  13AC303  is  RETURNING. 

TOT:  210803,  2  tanks  neutr,  1  a/c  lost. 

210809  :  TIME  UPDATE 

210810  :  TIME  UPDATE 

210812  :  TIME  UPOATE 

210812  :  Mission  H3AC203A  in  support  of  request  #3AC203  is  TAKING  OFF. 

210812  :  Mission  43AC304  in  support  of  request  #3AC304  is  RETURNING. 

TOT:  210807,  3  tanks  neutr,  1  a/c  lost. 

210813  :  TIME  UPDATE 

210813  :  Mission  43AC201A  recovered  at  BASE-6. 
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210813  :  TFU6  wilt  assign  only  4  of  the  8  planned  a/c  to  30M  alert  until  more  become  available. 


210814 

210814 

210815 

210816 

210816 

210818 

210818 

210818 

210818 

210818 

210818 

210818 

210818 

210821 

210822 

210822 

210822 

210823 

210823 

210823 

210824 

210824 

210824 

210824 

210826 

210826 


TIME  UPDATE 

Mission  #3AC202A  in  support  of  request  #3AC202  is  RETURNING. 

TOT:  210809,  4  tanks  neutr,  no  a/c  lost. 

TIME  UPDATE 

TIME  UPDATE 

Received  ASAP  request  13AC305  from  BLUE-DIV3-AL0  to  neutr  12  tanks,  NLT:  210930. 

TIME  UPDATE 

Mission  03AC3O2  recovered  at  BASE-2. 

TFW2  will  assign  only  6  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 

TFU2  will  assign  none  of  the  4  planned  a/c  to  30M  alert  until  more  become  available. 

Mission  #3AC303A  in  support  of  request  03AC3O3  is  RETURNING. 

TOT:  210813,  1  tanks  neutr,  no  a/c  lost. 

TFU2  will  assign  only  4  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 

TFU2  will  assign  none  of  the  4  planned  a/c  to  30M  alert  until  more  become  available. 

UOC  TFU2  ACCEPTED  request  #3AC305  &  tasked  the  following  mission: 

03AC3O5  •  2  F4s  from  BASE -2 

TIME  UPDATE 

TIME  UPDATE 

TFW3  will  assign  only  4  of  the  6  planned  a/c  to  30M  alert  until  more  become  available. 

UOC  TFU3  ACCEPTED  request  #3AC305  &  tasked  the  following  mission: 

83AC305A  -  2  A7s  from  BASE-3 

TIME  UPDATE 

Mission  83AC202  recovered  at  BASE-6. 

Mission  #3AC305  in  support  of  request  #3AC305  is  TAKING  OFF. 

TIME  UPDATE 

Mission  13AC101  recovered  at  BASE-1. 

TFW1  will  assign  only  3  of  the  4  planned  a/c  to  5M  alert  until  more  become  available. 
TFW1  will  assign  none  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 
TIME  UPDATE 

Mission  #3AC304  recovered  at  BASE-9. 
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210826  :  TFW9  will  assign  only  5  of  the  6  planned  a/c  to  30M  alert  until  More  become  available. 
210827  :  TIME  UPOATE 

210827  :  Mission  N3AC305A  in  support  of  request  #3AC305  is  TAKING  OFF. 

210827  :  Mission  13AC203  in  support  of  request  #3AC203  is  RETURNING. 

TOT:  210822,  6  tanks  neutr,  no  a/c  lost. 

210828  :  TIME  UPOATE 

210828  :  Mission  #3AC101A  in  support  of  request  03AC1O1  is  RETURNING. 

TOT:  210818,  6  tanks  neutr,  no  a/c  lost. 

210829  :  TIME  UPOATE 

210829  :  Mission  (F3AC303  recovered  at  BASE -2. 

210829  :  TFW2  will  assign  only  7  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 
210829  :  TFU2  will  assign  none  of  the  4  planned  a/c  to  30M  alert  until  more  become  available. 
210830  :  TIME  UPOATE 

210830  :  Received  ASAP  request  (F3AC103  from  BLUE-DI VI -ALO  to  neutr  S2  tanks,  NLT :  211020. 

210832  :  TIME  UPDATE 
210833  :  TIME  UPDATE 

210833  :  Mission  03AC2O2A  recovered  at  BASE-6. 

210833  :  TFW3  will  assign  none  of  the  6  planned  a/c  to  30M  alert  until  more  become  available. 

210833  :  WOC  TFW3  ACCEPTED  request  #3AC103  &  tasked  the  following  mission: 

#3AC103  -  4  A7s  from  BASE-3 

210833  :  Received  ASAP  request  #3AC204  from  BLUE-DIV2-ALO  to  neutr  20  tanks,  NLT:  211005. 

210833  :  TFW1  will  assign  only  4  of  the  8  planned  a/c  to  30M  alert  until  more  become  available. 

210833  :  WOC  TFU1  ACCEPTED  request  #3AC103  &  tasked  the  following  mission: 

#3AC103A  -  4  AlOs  from  BASE-1 

210833  :  Mission  83AC102  in  support  of  request  #3AC102  is  RETURNING. 

TOT:  210823,  28  tanks  neutr,  1  a/c  lost. 

210836  :  TIME  UPDATE 

210836  :  Received  ASAP  request  *3AC205  from  BLUE-DIV2-AL0  to  neutr  22  tanks,  NLT:  211010. 

210836  :  TFW10  will  assign  only  2  of  the  4  planned  a/c  to  5M  alert  until  more  become  available. 

210836  :  TFW10  will  assign  none  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 

210836  :  TFW10  will  assign  none  of  the  8  planned  a/c  to  30M  alert  until  more  become  available. 
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210836  :  UOC  TFU10  ACCEPTED  request  B3AC204  8  tasked  the  following  Missions: 

#3AC204  -  4  KARS  from  BASE -10 
03AC2O4A  -  2  HAAS  from  BASE-10 
B3AC204B  •  2  HARs  fro*  BASE -10 

210837  :  TIME  UPDATE 

210837  :  TFW6  will  assign  only  4  of  the  8  planned  a/e  to  30M  alert  until  More  become  available. 

210837  :  UOC  TFW6  ACCEPTED  request  B3AC205  l  tasked  the  following  Missions: 

B3AC205  -  4  FSs  from  BASE -6 
*  B3AC20SA  -  4  F5s  from  BASE -6 

210837  :  Mission  B3AC203A  in  support  of  request  B3AC203  is  RETURNING. 

TOT:  210832,  8  tanks  neutr,  no  a/c  lost. 

210838  :  TIME  UPDATE 

210838  :  Mission  #3AC103  in  support  of  request  B3AC103  is  TAKING  OFF. 

210839  :  TIME  UPDATE 

210839  :  Mission  B3AC303A  recovered  at  BASE-2. 

210839  :  TFW2  will  assign  only  3  of  the  4  planned  a/e  to  30M  alert  until  more  become  available. 
210840  :  TIME  UPDATE 
210841  :  TIME  UPDATE 

210841  :  Mission  «3AC204  in  support  of  request  B3AC204  is  TAKING  OFF. 

210842  :  TIME  UPDATE 

210842  :  Mission  B3AC203  recovered  at  BASE-10. 

210842  :  TFU10  will  assign  only  2  of  the  8  planned  a/c  to  15M  alert  until  More  became  available. 
210842  :  TFU10  will  assign  none  of  the  8  planned  a/c  to  30M  alert  until  More  become  available. 
210842  :  Mission  B3AC205  in  support  of  request  #3AC205  is  TAKING  OFF. 

210843  :  TIME  UPDATE 

210843  :  Mission  B3AC102A  in  support  of  request  B3AC102  is  RETURNING. 

TOT:  210833,  13  tanks  neutr,  no  a/c  lost. 

210844  :  TIME  UPDATE 

210844  :  Received  ASAP  request  B3AC104  from  8LUE-DIV1 -ALO  t6  neutr  50  tanks,  NLT:  211025. 

210845  :  TIME  UPDATE 
210846  :  TIME  UPDATE 

210846  :  Received  ASAP  request  B3AC306  from  BLUE-DIV3-AL0  to  neutr  10  tanks,  NLT:  211010. 

210848  :  TIME  UPDATE 
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210848  :  Received  ASAP  request  B3AC307  f row  BLUE-DIV3-ALO  to  neutr  13  tanks,  MLTs  211005. 

210849  :  TIME  UPOATE 
210850  :  TIME  UPDATE 

210850  :  Mission  #3AC101A  recovered  at  BASE-1. 

210850  :  TFW1  will  assign  only  4  of  the  8  planned  a/c  to  30M  alert  until  more  become  available. 

210850  :  UOC  TFW1  ACCEPTED  request  #3AC104  t  tasked  the  following  mission: 

B3AC104  -  4  AlOs  from  BASE-1 

210851  :  TIME  UPDATE 

210851  :  Mission  B3AC204A  in  support  of  request  B3AC204  is  TAKING  OFF. 

210851  :  Mission  03AC2O4B  in  support  of  request  (F3AC204  is  TAKING  OFF. 

210851  :  TFW3  will  assign  only  4  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 

210851  :  TFW3  will  assign  none  of  the  6  planned  a/c  to  30M  alert  until  more  become  available. 

210851  :  UOC  TFU3  ACCEPTED  request  B3AC104  &  tasked  the  following  mission: 

B3AC104A  -  4  A7s  from  BASE-3 

210852  :  TIME  UPDATE 

210852  :  TFU9  w-'il  assign  only  1  of  the  6  planned  a/c  to  30M  alert  until  more  became  available. 

210852  :  UOC  TFU9  ACCEPTED  request  B3AC306  &  tasked  the  following  mission: 

«3A  306  •  4  AJETs  from  BASE- 9 

210852  :  TFW9  wi  l  assign  only  5  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 

210852  :  TFW9  will  assign  none  of  the  6  planned  a/c  to  30M  alert  until  more  become  available. 

210852  :  UOC  TFUV  ACCEPTED  request  DBAC307  &  tasked  the  following  mission: 

«A'  307  -  4  AJETs  from  BASE -9 

210852  :  Mission  43AC203A  recovered  at  BASE-1n. 

210852  :  TFU10  will  assign  only  6  of  the  8  planned  a/c  to  15M  alert  until  store  become  available. 
210852  :  TFU10  will  assign  none  of  the  8  planned  e/c  to  30M  alert  until  more  become  available. 
210852  :  Mission  B3AC205A  in  support  of  request  B3AC205  is  TAKING  OFF. 

210853  :  TIME  UPDATE 
210854  :  TIME  UPOATE 

210854  :  Mission  H3AC305  in  support  of  request  B3AC305  is  RETURNING. 

TOT:  210849,  1  tanks  neutr,  no  a/c  lost. 
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210855  :  TIME  UPOATE 

210855  :  Mission  #3AC102  recovered  at  BASE-1. 

210855  :  TFW1  will  assign  only  2  of  the  8  planned  a/c  to  15M  alert  until  More  bee  owe  available. 
210856  :  TIME  UPDATE 

210856  :  Mission  #3AC104A  in  support  of  request  B3AC104  is  TAKING  OFF. 

210857  :  TIME  UPDATE 

210857  :  Mission  B3AC306  in  support  of  request  B3AC306  is  TAKING  OFF. 

210858  :  TINE  UPDATE 
210900  :  TIME  UPDATE 
210901  :  TIME  UPDATE 

210901  :  WOC  TFM3  ACCEPTED  request  #3AC104  &  tasked  the  following  mission: 

#3AC104B  -  2  A7s  from  BASE-3 

210902  :  TIME  UPDATE 
210903  :  TIME  UPOATE 

210903  :  Mission  B3AC103A  in  support  of  request  B3AC103  is  TAKING  OFF. 

210905  :  TIME  UPDATE 

210905  :  Mission  B3AC102A  recovered  at  BASE-1. 

210905  :  TFU1  will  assign  only  6  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 
210906  :  TIME  UPDATE 

210906  :  Mission  13AC305A  in  support  of  request  B3AC305  is  RETURNING. 

TOT:  210856,  5  tanks  neutr,  no  a/c  lost. 

210906  :  Mission  B3AC204  in  support  of  request  B3AC204  is  RETURNING. 

TOT:  210901,  A  tanks  neutr,  no  a/c  lost. 

210907  :  TIME  UPOATE 

210907  :  Mission  B3AC307  in  support  of  request  B3AC307  is  TAKING  OFF. 

210908  :  TIME  UPOATE 

210908  :  Received  ASAP  request  03AC2O6  from  BLUE-DIV2-AL0  to  neutr  16  tanks,  NLT :  211045. 

210910  :  TIME  UPDATE 
210911  :  TIME  UPOATE 

210911  :  Mission  B3AC205  in  support  of  request  03AC2O5  is  RETURNING. 

TOT:  210906,  5  tanks  neutr,  no  a/e  lost. 

210912  :  TIME  UPOATE 
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210915  :  TIME  UPDATE 


210915 

210916 

210916 

210916 

210916 

210916 

210916 

210917 

210917 

210917 

210917 

210917 

210920 

210920 

210920 

210921 

210921 

210921 

210921 

210921 

210921 

210922 

210922 

210922 


Mission  #3AC305  recovered  at  BASE -2. 

TIME  UPOATE 

Mission  #3AC204A  in  support  of  request  83AC204  is  RETURN I MG. 

TOT:  210911,  3  tanks  neutr,  no  a/c  lost. 

TFW10  will  assign  only  4  of  the  8  planned  a/c  to  15M  alert  until  aore  become  available. 

TFU10  will  assign  none  of  the  8  planned  a/c  to  SOM  alert  until  (tore  become  available. 

UOC  TFW10  ACCEPTED  request  (H3AC206  &  tasked  the  following  mission: 

#3AC206  -  2  HARs  from  BASE- 10 


Mission  #3AC204B  in  support  of  request  #3AC204  is  RETURNING. 

TOT:  210911,  2  tanks  neutr,  no  a/c  lost. 

TIME  UPDATE 

Mission  #3AC103  in  support  of  request  #3AC1J3  is  RETURNING. 

TOT:  210907,  9  tanks  neutr,  no  a/c  lost. 

TFU9  will  assign  only  1  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 

TFU9  will  assign  none  of  the  6  planned  a/c  to  30M  alert  until  more  become  available. 

UOC  TFU9  ACCEPTED  request  83AC206  t  tasked  the  following  mission: 

*3AC206A  -  4  AJETs  from  BASE -9 


TIME  UPDATE 

Mission  #3AC104  in  support  of  request  #3AC104  is  TAKING  OFF. 

Received  ASAP  request  #3AC207  from  BLUE-DIV2-ALO  to  neutr  17  tanks,  NLT:  211050. 

TIME  UPDATE 

Mission  83AC204  recovered  at  BASE-10. 

TFU10  will  assign  none  of  the  8  planned  a/c  to  30M  alert  until  store  become  available. 

Mission  83AC205A  in  support  of  request  #3AC205  is  RETURNING. 

TOT:  210916,  5  tanks  neutr,  no  a/c  lost. 

Mission  83AC306  in  support  of  request  03AC3O6  is  RETURNING. 

TOT:  210916,  6  tanks  neutr,  1  a/c  lost. 

Mission  #3AC206  in  support  of  request  83AC206  is  TAKING  OFF. 

TIME  UPDATE 

TFW9  will  assign  only  3  of  the  4  planned  s/c  to  5M  alert  until  more  become  available. 
TFW9  will  assign  none  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 
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210922 

210922 

210922 

21092S 

210926 

210926 

210926 

210927 

210930 

210930 

210930 

210930 

210930 

210931 

210931 

210931 

210931 

210931 

210931 

210931 

210931 

210931 

210931 

210931 

210932 
210932  : 
210932  : 


:  TFU9  wilt  assign  none  of  the  6  planned  a/c  to  30H  alert  until  more  become  available. 

:  U0C  TFU9  ACCEPTED  request  B3AC207  &  tasked  the  following  Mission: 

#SAC207  -  2  AJETs  from  BASE -9 

:  Mission  S3AC206A  in  support  of  request  #3AC206  is  TAKING  OFF. 

:  TINE  UPDATE 
:  TINE  UPOATE 

:  Received  ASAP  request  B3AC308  from  8LUE*D!V3-ALO  to  neutr  20  tanks,  NLT:  211035. 

:  Received  ASAP  request  #3AC105  from  BLUE-DIV1-ALO  to  neutr  61  tanks,  NLT:  211050. 

:  TINE  UPDATE 
TINE  UPOATE 

Mission  #3AC305A  recovered  at  BASE-3. 

TFU3  will  assign  only  6  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 
TFU3  will  assign  none  of  the  6  planned  a/c  to  30M  alert  until  more  become  available. 
Mission  #3AC205  recovered  at  BASE-6. 

TIME  UPDATE 

TFU3  will  assign  only  2  of  the  4  planned  a/c  to  5M  alert  until  more  became  available. 

TFU3  will  assign  none  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 

TFU3  will  assign  none  of  the  6  planned  a/c  to  30M  alert  until  more  become  available. 

UOC  TFU3  ACCEPTED  request  B3AC105  l  tasked  the  following  missions: 

X3AC105  -  4  A7s  from  BASE -3 
«3AC105A  •  4  A7s  from  BASE -3 

Mission  B3AC104B  in  support  of  request  #3AC104  is  TAKING  OFF. 

Mission  03AC2O4A  recovered  at  BASE-10. 

TFU10  will  assign  only  2  of  the  8  planned  a/c  to  30M  alert  until  more  become  available. 
Mission  B3AC204B  recovered  at  BASE-10. 

TFW10  will  assign  only  4  of  the  8  planned  a/c  to  30M  alert  until  more  become  available. 

Mission  B3AC307  in  support  of  request  #3AC307  is  RETURNING. 

TOT:  210926,  5  tanks  neutr,  no  a/c  lost. 

TIM  UPOATE 

Received  ASAP  request  «3AC208  from  BLUE-DIV2-ALO  to  neutr  14  tanks,  NLT:  211110. 

UOC  TFU3  ACCEPTED  request  63AC308  l  tasked  the  following  mission: 

B3AC308  -  2  A7s  from  BASE-3 
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210932  :  Mission  #3AC207  in  support  of  request  #3AC207  is  TAKING  OFF. 

210932  :  WOC  TFW9  ACCEPTED  request  #3AC308  &  tasked  the  following  mission: 

03AC3O8A  -  2  AJETs  from  BASE -9 

210934  :  TIME  UPDATE 

210934  :  Received  ASAP  request  #3AC309  from  BLUE-DIV3-ALO  to  neutr  27  tanks,  NIT:  211050. 

210935  :  TIME  UPDATE 
210935  :  TIME. UPDATE 

210935  :  Mission  #3AC306  recovered  at  BASE-9. 

210935  :  TFW9  will  assign  only  2  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 

210935  :  TFW9  will  assign  none  of  the  6  planned  a/c  to  30M  alert  until  more  become  available. 

210935  :  Mission  #3AC104A  in  support  of  request  03AC1O4  is  RETURNING. 

TOT:  210925,  6  tanks  neutr,  no  a/c  lost. 

210936  :  TIME  UPDATE 

210936  :  Mission  #3AC105  in  support  of  request  03AC1O5  is  TAKING  OFF. 

210937  :  TIME  UPDATE 
210940  :  TIME  UPDATE 

210940  :  Mission  #3AC205A  recovered  at  BASE-6. 

210940  :  Mission  #3AC103A  in  support  of  request  #3AC103  is  RETURNING. 

TOT:  210930,  30  tanks  neutr,  1  a/c  lost. 

210941  :  TIME  UPDATE 

210941  :  Mission  R3AC103  recovered  at  BASE-3. 

210941  :  TFW3  will  assign  only  2  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 

210941  :  TFU3  will  assign  none  of  the  6  planned  a/c  to  30M  alert  until  more  become  available. 

210943  :  TIME  UPDATE 

210945  :  TIME  UPDATE 

210945  :  TFU2  will  assign  only  1  of  the  4  planned  a/c  to  SOM  alert  until  more  become  available. 

210945  :  WOC  TFW2  ACCEPTED  request  #3AC208  &  tasked  the  following  missions: 

#3AC208  -  2  F4s  from  BASE -2 
#3AC208A  -  2  F4s  from  BASE -2 

210945  :  Mission  B3AC307  recovered  at  BASE-9. 

210945  :  TFW9  will  assign  only  6  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 
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210945  :  TFW9  Mill  assign  non*  of  the  6  planned  a/c  to  30M  alert  until  store  become  available. 
210946  :  TIME  UPDATE 

210946  :  Mission  83AC105A  in  support  of  request  #3AC105  is  TAKING  OFF. 

210946  :  Mission  #3AC206  in  support  of  request  #3AC206  is  RETURNING. 

TOT:  210941,  2  tanks  neutr,  no  a/c  lost. 

210946  :  Mission  #3AC206A  in  support  of  request  83AC206  is  RETURNING. 

TOT:  210941,  2  tanks  neutr,  no  a/c  lost. 

210947  :  TIME  UPDATE 

210947  :  WOC  TFW1  REFUSED  request  43AC309  which  was  for  4  AlOs  NLT:  211050. 

REASON:  *  NOT  ENOUGH  AIRCRAFT  AVAILABLE  TO  MEET  THE  NLT 
210948  :  TIME  UPDATE 

210948  :  Received  ASAP  request  «3AC310  from  BLUE-DIV3-AL0  to  neutr  19  tanks,  NLT:  211115. 
210948  :  WOC  TFU1  REFUSED  request  #3AC309  which  was  for  4  AlOs  NLT:  211050. 

REASON:  *  NOT  ENOUGH  AIRCRAFT  AVAILABLE  TO  MEET  THE  NLT 
210951  :  TIME  UPOATE 

210951  :  WOC  TFW3  REFUSED  request  H3AC309  which  was  for  4  A7s  NLT:  211050. 

REASON:  *  NOT  ENOUGH  AIRCRAFT  AVAILABLE  TO  MEET  THE  NLT 
210956  :  TIME  UPDATE 

210956  :  Mission  #3AC207  in  si^jport  of  request  83AC207  is  RETURNING. 

TOT:  210951,  1  tanks  neutr,  no  a/c  lost. 

210957  :  TIME  UPDATE 

210957  :  WOC  TFW6  REFUSED  request  #3AC309  which  was  for  4  F5s  NLT:  211050. 

REASON:  *  NOT  ENOUGH  AIRCRAFT  AVAILABLE  TO  MEET  THE  NLT 

210957  :  Mission  03AC1O4  in  support  of  request  83AC104  is  RETURNING. 

TOT:  210947,  35  tanks  neutr,  2  a/c  lost. 

210957  :  WOC  TFW6  ACCEPTED  request  83AC309  &  tasked  the  following  missions: 

83AC309  -  2  F5s  from  BASE-6 
83AC309A  -  2  F5s  from  BASE-6 

210958  :  TIME  UPOATE 
210959  :  TIME  UPDATE 

210959  :  Mission  I3AC104A  recovered  at  BASE-3. 

210959  :  TFW3  will  assign  only  6  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 
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210959  :  TFW3  will  assign  none  of  the  6  planned  a/c  to  30N  alert  until  more  become  available. 
210959  :  WOC  TFU10  REFUSED  request  #3AC310  which  was  for  4  HARs  NLT:  211115. 

REASON:  *  NOT  ENOUGH  AIRCRAFT  AVAILABLE  TO  MEET  THE  NLT 
211000  :  TIME  UPDATE 

211000  :  Mission  03AC2O6A  recovered  at  BASE-9. 

211000  :  TFW9  will  assign  only  2  of  the  6  planned  a/c  to  30M  alert  until  more  become  available. 

211000  :  TFU6  wilt  assign  only  4  of  the  8  planned  a/c  to  30M  alert  until  more  become  available. 

211000  :  WOC  TFU6  ACCEPTED  request  #3AC310  &  tasked  the  following  missions: 

03AC31O  -  2  F5s  from  BASE-6 
« 3AC310A  -  2  F5s  from  BASE -6 

211001  :  TIME  UPDATE 

211001  :  Mission  #3AC206  recovered  at  BASE-10. 

211001  :  TFW10  will  assign  only  6  of  the  8  planned  a/c  to  30M  alert  until  more  become  available. 
211002  :  TIME  UPDATE 

211002  :  Mission  #3AC103A  recovered  at  BASE-1. 

211002  :  TFU1  will  assign  only  7  of  the  8  planned  a/c  to  30M  alert  until  more  become  available. 
211002  :  Mission  #3AC308  in  support  of  request  #3AC308  is  TAKING  OFF. 

211002  :  Mission  #3AC308A  in  support  of  request  #3AC308  is  TAKING  OFF. 

211005  :  TIME  UPDATE 
211006  :  TIME  UPOATE 
211007  :  TIME  UPDATE 
211008  :  TINE  UPOATE 
211009  :  TIME  UPDATE 
211010  :  TIME  UPDATE 

211010  :  Mission  #3AC207  recovered  at  BASE-9. 

211010  :  TFW9  will  assign  only  4  of  the  6  planned  a/c  to  30M  alert  until  more  become  available. 

211010  :  Mission  #3AC104B  in  support  of  request  03AC1O4  is  RETURNING. 

TOT:  211000,  2  tanks  neutr,  no  a/c  lost. 

211011  :  TIME  UPDATE 

211012  :  TIME  UPOATE 

211013  :  TIME  UPOATE 
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211013  :  Received  ASAP  request  B3AC209  from  BLUE-DIV2-ALO  to  neutr  20  tanks,  NLT:  211140. 

211015  :  TINE  UPDATE 

211015  :  Received  JAAT  request  B3AC106  from  BLUE-D1V1-ALO,  TOT:  211215  NLT:  211315. 

211015  :  mission  #3 AC 105  in  support  of  request  B3AC105  is  RETURNING. 

TOT:  211005,  7  tanks  neutr,  no  a/c  lost. 

211016  :  TINE  UPDATE 

211017  :  TINE  UP0ATE 

211019  :  TINE  UPDATE 

211019  :  TFU6  will  assign  none  of  the  8  planned  a/c  to  30N  alert  until  more  became  avei table. 

211019  :  WOC  TFW6  ACCEPTED  request  B3AC209  &  tasked  the  following  missions: 

R3AC209  •  2  F5s  from  BASE-6 
B3AC209A  -  2  F5s  from  BASE -6 

211019  :  mission  B3AC104  recovered  at  BASE-1. 

211021  :  TINE  UPDATE 

211021  :  mission  B3AC309  in  support  of  request  B3AC309  is  TAKING  OFF. 

211021  :  TFW1  will  assign  only  5  of  the  8  planned  a/c  to  3 ON  alert  until  more  become  available. 

211021  :  WOC  TFU1  ACCEPTED  request  *3AC106  t  tasked  the  following  mission: 

B3AC106  -  4  AlOs  from  BASE-1 

211022  :  TINE  UPDATE 
211023  :  TINE  UPOATE 

211023  :  Received  ASAP  request  B3AC107  from  BLUE-D1V1-AL0  to  neutr  61  tanks,  NLT:  211205. 

211024  :  TINE  UPDATE 
211025  :  TINE  UPDATE 

211025  :  mission  B3AC105A  in  support  of  request  B3AC105  is  RETURNING. 

TOT:  211015,  5  tanks  neutr,  1  a/c  lost. 

211026  :  TINE  UPOATE 

211026  :  mission  B3AC308A  in  support  of  request  B3AC308  is  RETURNING. 

TOT:  211021,  2  tanks  neutr,  no  a/c  lost. 

211026  :  TFW1  will  assign  only  2  of  the  8  plat-vied  a/c  to  15N  alert  until  more  become  available. 

211026  :  U0C  TFW1  ACCEPTED  request  B3AC106  l  tasked  the  following  mission: 

I3AC106A  -  4  AlOs  from  BASE-1 

211027  :  TINE  UPOATE 


76 


211027  :  TFW9  will  assign  only  2  of  the  6  planned  a/c  to  30M  alert  until  more  become  available. 


211027  :  WOC  TFW9  ACCEPTED  request  03AC1O7  t  tasked  the  following  missions: 
#3AC107  •  2  AJETs  from  BASE -9 
43AC107A  -  2  AJETs  from  BASE-9 


211027  :  WOC  TFW3  ACCEPTED  request  13AC107  &  tasked  the  following  mission: 

#3AC107B  -  2  A7s  from  BASE -3 

211027  :  Mission  03AC3O9A  in  support  of  request  03AC3O9  is  TAKING  OFF. 

211028  :  TIME  UPDATE 

211028  :  Mission  N3AC310  in  support  of  request  #3AC31G  is  TAKING  OFF. 

211029  :  TIME  UPDATE 
211030  :  TIME  UPDATE 
211031  :  TIME  UPDATE 

211031  :  Mission  N3AC208  in  support  of  request  #3AC208  is  TAKING  OFF. 

211032  :  TIME  UPDATE 

211032  :  Received  ASAP  request  #3AC210  from  BLUE-DIV2-ALO  to  neutr  26  tanks,  NLT:  211210. 

211033  :  TIME  UPDATE 

211033  :  Mission  #3AC208A  in  support  of  request  N3AC208  is  TAKING  OFF. 

211033  :  TFW6  will  assign  only  4  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 

211033  :  TFW6  will  assign  none  of  the  8  planned  a/c  to  30M  alert  until  more  be~cme  available. 

211033  :  WOC  TFW6  ACCEPTED  request  #3AC210  &  tasked  the  following  missions: 

(F3AC210  -  2  F5s  from  BASE-6 
N3AC210A  -  2  F5s  from  BASE-6 

211034  :  TIME  UPDATE 

211034  :  Mission  N3AC104B  recovered  at  BASE-3. 

211035  :  TIME  UPDATE 
211036  :  TIME  UPOATE 

211036  :  Received  ASAP  request  #3AC311  from  BLUE-DIV3-AIO  to  neutr  17  tanks,  NLT:  211145. 

211 037  :  TIME  UPOATE 

211038  :  TIME  UPOATE 

211038  :  TFW2  will  assign  only  5  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 

211038  :  TFW2  will  assign  none  of  the  4  planned  a/c  to  30M  alert  until  more  become  available. 
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211038  :  WOC  TFW2  ACCEPTED  request  B3AC311  ft  tasked  the  following  missions: 

#3AC311  -  2  F4s  from  BASE-2 
«AC311A  •  2  F4s  from  BASE-2 

211038  :  Mission  #3AC310A  in  support  of  request  03AC31O  is  TAKING  OFF. 

211038  :  Received  ASAP  request  #3AC108  from  BIUE-DIV1-AIO  to  neutr  72  tanks,  NLT :  211215. 

211039  :  TIME  UPDATE 

211039  :  Mission  N3AC105  recovered  at  BASE-3. 

211039  :  Tf*J3  will  assign  only  2  of  the  6  planned  a/c  to  30M  alert  until  more  become  available. 

211039  :  TFW1  will  assign  only  2  of  the  4  planned  a/c  to  5M  alert  until  more  become  available. 

211039  :  TFW1  will  assign  none  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 

211039  :  WOC  TFW1  ACCEPTED  request  03AC1O8  &  tasked  the  following  missions: 

N3AC108  -  2  AlOs  from  BASE-1 
«3AC108A  -  2  AlOs  from  BASE-1 

211039  :  Mission  B3AC209  in  support  of  request  #3AC209  is  TAKING  OFF. 

211039  :  TFU10  will  assign  only  2  of  the  8  planned  a/c  to  30M  alert  until  more  become  available. 

211039  :  WOC  TFU10  ACCEPTED  request  #3AC108  &  tasked  the  following  missions: 

*3AC108B  •  2  HARs  from  BASE-10 
B3AC108C  •  2  HARs  from  BASE- 10 

211040  :  TIME  UPDATE 

211040  :  Mission  N3AC308A  recovered  at  BASE-9. 

211041  :  TIME  UPDATE 

211041  :  Mission  N3AC308  in  support  of  request  N3AC308  is  RETURNING. 

TOT:  211031,  4  tanks  neutr,  no  a/c  lost. 

211043  :  TIME  UPDATE 

211043  :  Mission  (F3AC311  in  support  of  request  N3AC311  is  TAKING  OFF. 

211044  :  TIME  UPDATE 
211045  :  TIME  UPDATE 

211045  :  Mission  B3AC107  in  support  of  request  #3AC107  is  TAKING  OFF. 

211046  :  TIM  UPDATE 

211046  :  Mission  #3AC108  in  support  of  request  B3AC108  is  TAKING  OFF. 

211047  :  TIME  UPDATE 
211048  :  TIME  UPDATE 


211048  :  Mission  03AC311A  in  support  of  request  #3AC311  is  TAKING  OFF. 

211049  :  TIME  UPDATE 

211049  :  Mission  #3AC105A  recovered  at  BASE-3. 

211049  :  TFW3  will  assign  only  5  of  the  6  planned  a/c  to  30M  alert  until  more  become  available. 
211050  :  TIME  UPDATE 

211050  :  Mission  #3AC309  in  support  of  request  #3AC309  is  RETURNING. 

TOT:  211045,  2  tanks  neutr,  no  a/c  lost. 

211051  :  TIME  UPCATE 

211052  :  TIME  UPDATE 

211053  :  TIME  UPDATE 

211053  :  Mission  13AC209A  in  support  of  request  #3AC209  is  TAKING  OFF. 

211054  :  TIME  UPDATE 
211055  :  TIME  UPDATE 
211056  :  TIME  UPDATE 

211056  :  Mission  #3AC309A  in  support  of  request  #3AC309  is  RETURNING. 

TOT:  211051,  2  tanks  neutr,  1  a/c  lost. 

211057  :  TIME  UPDATE 

211057  :  Received  ASAP  request  03AC312  from  BLUE-DIV3-ALO  to  neutr  23  tanks,  NLT:  211215. 

211057  :  Mission  #3AC107A  in  support  of  request  #3AC107  is  TAKING  OFF. 

211057  :  Mission  #3AC107B  in  support  of  request  IK3AC107  is  TAKING  OFF. 

211057  :  'lission  #3AC310  in  support  of  request  #3AC310  is  RETURNING. 

TOT:  211052,  2  tanks  neutr,  no  a/c  lost. 

211058  :  TIME  UPDATE 

211059  :  TIME  UPDATE 

211059  :  TFU2  will  assign  only  1  of  the  4  planned  a/c  to  5M  alert  until  more  become  available. 

211059  :  TFW2  will  assign  none  of  the  8  planned  a/c  to  15M  alert  until  more  become  available. 

211059  :  TFW2  will  assign  none  of  the  4  planned  a/c  to  30M  alert  until  more  become  available. 

211059  :  WOC  TFW2  ACCEPTED  request  #3AC312  &  tasked  the  following  missions: 

#3AC312  -  2  F4s  from  BASE-2 
03AC312A  -  2  F4s  from  BASE-2 
#3AC312B  •  2  F4s  from  BASE -2 
*3AC312C  •  2  F4s  from  BASE-2 

211059  :  Mission  #3AC210  in  support  of  request  #3AC210  is  TAKING  OFF. 
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211100  :  TINE  UPDATE 


211101  :  TINE  UPDATE 
211102  :  TINE  UPDATE 

211102  :  Nits I on  N3AC210A  In  support  of  request  #3AC210  (s  TAKING  OFF. 

211103  :  TINE  UPOATE 

211103  :  Nission  #3AC208  in  support  of  request  #3AC208  is  RETURNING. 

TOT:  211058,  2  tanks  neutr,  no  s/e  lost. 

211104  :  TINE  UPOATE 

211104  :  Nission  #3AC312  in  support  of  request  #3AC312  is  TAKING  OFF. 

211105  :  TINE  UPDATE 

211105  :  Nission  N3AC308  recovered  at  BASE-3. 

211105  :  Nission  #3AC208A  in  support  of  request  #3AC208  is  RETURNING. 

TOT:  211100,  1  tanks  neutr,  no  a/c  lost. 

211106  :  TINE  UPDATE 

211106  :  Nission  «3AC108B  in  support  of  request  #3AC108  is  TAKING  OFF. 

211107  :  TINE  UPDATE 

211107  :  Nission  #3AC310A  in  support  of  request  #3AC310  is  RETURN I Nr 
TOT:  211102,  3  tanks  neutr,  no  a/c  lost. 

211108  :  TINE  UPDATE 

211108  :  Nission  #3AC108C  in  support  of  request  #3AC108  is  TAKING  OFF. 

211108  :  Nission  N3AC209  in  support  of  request  N3AC209  is  RETURNING. 

TOT:  211103,  4  tanks  neutr,  no  a/c  lost. 

211109  :  TINE  UPOATE 

211109  :  Nission  N3AC309  recovered  at  BASE-6. 

211109  :  TFW6  will  assign  only  6  of  the  8  planned  a/c  to  15N  alert  until  more  become  available. 

211109  :  TFW6  will  assign  none  of  the  8  planned  a/c  to  30N  atert  until  more  become  available. 

211109  :  Nission  N3AC312A  in  si^port  of  request  N3AC312  is  TAKING  OFF. 

211109  :  Nission  #3AC107  in  support  of  request  #3AC107  is  RETURNING. 

TOT:  211104,  2  tanks  neutr,  1  a/c  lost. 
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Expert  Evaluation 
9/13/89 


SUPERKEATS  EVALUATION 


SuperKEATS  was  designed  as  a  simulation  environment  to  allow  the  Fighter  Duty 
Officer  (FDO)  in  the  712th  ASOC  to  make  decisions  about  Close  Air  Support 
(CAS) .  The  primary  goal  in  using  SuperKEATS  is  to  provide  decision-making 
training  to  the  FDO.  Below  we  would  like  you  to  evaluate  the  quality  of  the 
simulation  environment  and  training  capability  of  SuperKEATS. 

I .  SIMULATION  ENVIRONMENT 

Rate  the  simulation  environment  in  SuperKEATS  based  on  the  parameters  below. 
Please  explain  your  rating  by  answering  the  questions  and  adding  any  additional 
comments  you  care  to. 

[Rating  scale:  1  to  5  where  1  -  poor/disagree/no  and  5  -  excellent/agree/yes] 


A.  Completeness  of  the  representation  -  overall  rating: 

SuperKEATS  attempts  to  reduce  the  ASOC  environment  to 
just  those  elements  required  fc»*  the  task  at  hand  -- 
decision-making  about  CAS. 

1.  Were  all  the  objects  (people,  organizations,  things) 
that  you  would  expect  to  see  included  in  the 
simulation?  If  not,  what  was  missing?  Please 
indicate  the  importance  of  the  missing  items, 
(rating:  1  -  low  importance,  5  -  very  important) 


2.  Were  all  the  types  of  events  that  you  might  expect 
to  encounter  included  in  the  simulation?  If  not, 
which  event  types  were  missing?  Please  rate  missing 
types  as  to  importance . 

(rating:  1  -  low  importance,  5  -  very  important) 
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Non-worthwhile  targets  --  need  to  train  FDO  to  think 
about  target  types  --  and  reject/disapprove. 
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3. 


4. 


B. 


1. 


2. 


Were  you  able  to  access  all  the  information  you 
thought  you  should?  If  not,  what  was  missing  and 
please  rate  its  importance. 

Need  quick  access  to  available  aircraft  for  tasking. 
Put  column  on  Status  Board. 


Were  you  able  to  communicate  all  the  information  you 
needed  to  tell?  If  not,  what  was  missing  and  please 
rate  its  importance . 

Cannot  communicate  with  G-3  Air  concerning  priorities 
and  possible  need  to  change. 


Correctness  of  the  simulation  -  overall  rating: 

D? d  the  objects  in  the  simulation  react  as  you 
would  expect  them  to  in  a  real  situation? 

If  not,  how  did  they  differ  and  how  important 
is  the  difference? 

The  ground  force  losses  were  instantly  replaced  - 
Do  not  think  that  replacement  armor  avail 
reserves  -  (at  least  not  during  course  of  day) 
if  major  replacements  needed,  that  unit  would 
be  replaced.  This  did  not  allow  FDO  to  change 
focus  to  another  area  -  -  too  many  tanks/APCs 
encountered/destroyed  in  given  area. 


In  KEATS  we  tried  to  make  things  look  and  work 
the  same  as  in  the  ASOC  environment  --  e.g., 
status  boards,  briefings,  and  air  request  8001 
forms.  In  SuperKEATS  we  tried  to  retain  the 
functionality  but  let  the  forms  and  status  boards 
be  implemented  in  manner  more  natural  to  the 
computer  terminal.  Did  you  find  this  functional 
representation  in  SuperKEATS  to  be  easily  understood 
coming  from  the  "real  world"?  If  not,  what  were  the 
problems  and  how  did  they  distort  or  confuse  your 
view  of  what  was  going  on? 
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C.  Usable  simulation  -  overall  rating: 

1.  Did  the  Top  Bar  function  selection  allow  you  to 
easily  take  the  required  actions?  If  not,  what 
were  the  problems  in  using  the  Top  Bar? 

Most  of  time  took  too  long  and  put  display  over 
something  else  needed  --  much  more  functional 
to  create  windows,  size,  and  place  where  easily/ 
quickly  accessible . 

2.  Did  the  user  configurable  screen  with  windows 
allow  you  to  easily  set  up  and  access  information 
when  you  needed  it?  If  not,  what  problems  did 

you  have?  Do  you  have  suggestions  to  make  it  easier 
to  use? 

Could  get  very  busy  at  times  --  still  recommend 
dual  screen  or  larger  display. 


3.  Was  the  system  response  time  to  user  input  actions 
reasonably  fast?  If  not,  which  actions  were  distur 
bingly  slow? 

As  the  game  progressed ,  the  responses  slowed. 
However,  this  not  totally  unrealistic  because 
real-world  ADP  speed  slows  as  the  battle 
tempo  increases. 


II.  DECISION-MAKING  TRAINING 


A.  Coverage  of  material  -  overall  rating: 

1.  Are  all  the  important  areas  of  decision-making 

training  for  a  FDO  doing  CAS  covered  in  SuperKEATS? 
If  not,  please  indicate  the  missing  areas  that 
should  be  Included. 

a.  Need  events/requests  that  would  be  disapproved. 

b.  Need  actions  to  have  and  not  have  positive 
effects  on  war  in  particular  division  sectors 
(allow  shifting  of  emphasis /concern) . 

c.  Take  out  a  base  for  an  hour  (a  period  of  time 
that  will  effect  decisions  on  tasking/servicing 
requests) . 
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B.  Validity  of  the  training  -  overall  rating: 

1.  Does  what  a  novice  learns  about  FDO  decision¬ 
making  from  using  SuperKEATS  represent  what  you 
would  expect  them  to  use  in  the  ASOC?  If  not, 
how  does  it  differ? 


C.  Efficiency  of  training  -  overall  rating: 

1 .  Is  the  time  spent  running  a  SuperKEATS  game  well 
used?  For  a  novice?  For  an  expert? 

How  could  it  be  changed  to  provide  more  training 
in  less  time? 

Could  be  improved,  of  course.  Okay  now  for  novice 
(with  some  additions  previously  listed). 

Some  more  complex  events  could  be  generated  for 
the  experienced  FDO. 


D.  Usefulness  of  suggestions/options 

1.  Did  you  feel  that  tasking  suggestions/tasking 
option  was  a  useful  feature  for  training? 

If  not,  what  are  your  objections  and/or  how 
would  you  have  done  it  differently? 

Yes,  for  the  novice.  And  it  could  be  a 
useful  tool  for  real-world  tasking. 


E.  Usefulness  of  tasking  feedback 

1.  Did  you  feel  that  the  optional  feedback  to  tasking 
was  a  useful  feature  for  training.  If  not,  why  not 
and/or  how  might  it  have  been  done  differently? 
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Good  for  the  nov ice/ learner . 

May  for  complex  events  for  the  experienced  if 
using  for  review  and  let  him  know  if  all 
areas  considered. 
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IV.  GENERAL  EVALUATION  AND  COMMENTS 


A.  Do  you  see  uses  for  SuperKEATS  besides  training?  If  so,  please 
elaborate . 

Would  require  me Jor  work  to  interface  wLth  other  systems  if 
want  something  operational.  Those  efforts  are  already  in 
progress  --  maybe  a  tie  to  other  simulation  systems. 

Again,  for  training. 


B.  What  are  the  features  that  you  like  most  about  SuperKEATS?  How  could 
they  be  made  even  better? 


Action  --  Reaction  --  Result. 


C.  What  are  the  features  that  you  like  least  about  SuperKEATS?  Could  they 
be  changed  to  improve  its  usefulness?  If  so,  how? 

Some  displays  could  be  changed  to  be  more  functional 
and  complete  with  information. 

User  Build  Mode?  Is  there  one?  Needs  to  be. 


D.  Rate  SuperKEATS  in  its  present  form  in  terms  of 
usefulness  to  the  ASOC. 


E.  Rate  SuperKEATS  modified  as  you  specify  (here  or 
as  in  B  and  C  above)  in  terms  of  its  potential 
usefulness  to  the  ASOC. 
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